[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260807 11:08:22.603708 24034 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.120.190:46829
I20260807 11:08:22.606045 24034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:08:22.607699 24034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:08:22.648424 24041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:08:24.038340 24040 debug-util.cc:398] Leaking SignalData structure 0x7b080000e040 after lost signal to thread 24034
W20260807 11:08:22.648424 24042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:08:24.835873 24034 thread.cc:636] OpenStack (cloud detector) Time spent creating pthread: real 2.194s user 0.830s sys 1.358s
W20260807 11:08:24.836496 24034 thread.cc:602] OpenStack (cloud detector) Time spent starting thread: real 2.195s user 0.830s sys 1.358s
W20260807 11:08:24.845729 24044 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 2190 milliseconds
W20260807 11:08:24.846582 24045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:08:24.846684 24034 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260807 11:08:24.848249 24034 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 11:08:24.848562 24034 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 11:08:24.848749 24034 hybrid_clock.cc:648] HybridClock initialized: now 1786100904848732 us; error 0 us; skew 500 ppm
I20260807 11:08:24.850013 24034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:08:24.859869 24034 webserver.cc:460] Webserver started at http://127.23.120.190:42653/ using document root <none> and password file <none>
I20260807 11:08:24.861377 24034 fs_manager.cc:359] Metadata directory not provided
I20260807 11:08:24.861649 24034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:08:24.862530 24034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:08:24.871995 24034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786100902301536-24034-0/minicluster-data/master-0-root/instance:
uuid: "0244322652a34ae4b3b7e3b01e7ec2b2"
format_stamp: "Formatted at 2026-08-07 11:08:24 on dist-test-slave-g170"
I20260807 11:08:24.884227 24034 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.005s sys 0.005s
I20260807 11:08:24.894631 24056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:08:24.896559 24034 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.008s
I20260807 11:08:24.897107 24034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786100902301536-24034-0/minicluster-data/master-0-root
uuid: "0244322652a34ae4b3b7e3b01e7ec2b2"
format_stamp: "Formatted at 2026-08-07 11:08:24 on dist-test-slave-g170"
I20260807 11:08:24.897507 24034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786100902301536-24034-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786100902301536-24034-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786100902301536-24034-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:08:24.964249 24034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:08:24.967207 24034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:08:24.968052 24034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:08:25.224077 24034 rpc_server.cc:307] RPC server started. Bound to: 127.23.120.190:46829
I20260807 11:08:25.224398 24122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.120.190:46829 every 8 connection(s)
I20260807 11:08:25.241575 24123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:25.289031 24123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0244322652a34ae4b3b7e3b01e7ec2b2: Bootstrap starting.
I20260807 11:08:25.304529 24123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0244322652a34ae4b3b7e3b01e7ec2b2: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:25.314088 24123 log.cc:824] T 00000000000000000000000000000000 P 0244322652a34ae4b3b7e3b01e7ec2b2: Log is configured to *not* fsync() on all Append() calls
I20260807 11:08:25.330513 24123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0244322652a34ae4b3b7e3b01e7ec2b2: No bootstrap required, opened a new log
I20260807 11:08:25.400678 24123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0244322652a34ae4b3b7e3b01e7ec2b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0244322652a34ae4b3b7e3b01e7ec2b2" member_type: VOTER }
I20260807 11:08:25.401638 24123 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0244322652a34ae4b3b7e3b01e7ec2b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:25.402057 24123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0244322652a34ae4b3b7e3b01e7ec2b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0244322652a34ae4b3b7e3b01e7ec2b2, State: Initialized, Role: FOLLOWER
I20260807 11:08:25.403959 24123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0244322652a34ae4b3b7e3b01e7ec2b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0244322652a34ae4b3b7e3b01e7ec2b2" member_type: VOTER }
I20260807 11:08:25.405336 24123 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0244322652a34ae4b3b7e3b01e7ec2b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:08:25.405786 24123 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0244322652a34ae4b3b7e3b01e7ec2b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:08:25.406380 24123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0244322652a34ae4b3b7e3b01e7ec2b2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:25.414229 24123 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0244322652a34ae4b3b7e3b01e7ec2b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0244322652a34ae4b3b7e3b01e7ec2b2" member_type: VOTER }
I20260807 11:08:25.415503 24123 leader_election.cc:302] T 00000000000000000000000000000000 P 0244322652a34ae4b3b7e3b01e7ec2b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0244322652a34ae4b3b7e3b01e7ec2b2; no voters:
I20260807 11:08:25.419456 24123 leader_election.cc:288] T 00000000000000000000000000000000 P 0244322652a34ae4b3b7e3b01e7ec2b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:08:25.420316 24127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0244322652a34ae4b3b7e3b01e7ec2b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:08:25.429638 24127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0244322652a34ae4b3b7e3b01e7ec2b2 [term 1 LEADER]: Becoming Leader. State: Replica: 0244322652a34ae4b3b7e3b01e7ec2b2, State: Running, Role: LEADER
I20260807 11:08:25.432348 24127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0244322652a34ae4b3b7e3b01e7ec2b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0244322652a34ae4b3b7e3b01e7ec2b2" member_type: VOTER }
I20260807 11:08:25.434818 24123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0244322652a34ae4b3b7e3b01e7ec2b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:08:25.459697 24128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0244322652a34ae4b3b7e3b01e7ec2b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0244322652a34ae4b3b7e3b01e7ec2b2. Latest consensus state: current_term: 1 leader_uuid: "0244322652a34ae4b3b7e3b01e7ec2b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0244322652a34ae4b3b7e3b01e7ec2b2" member_type: VOTER } }
I20260807 11:08:25.459605 24129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0244322652a34ae4b3b7e3b01e7ec2b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0244322652a34ae4b3b7e3b01e7ec2b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0244322652a34ae4b3b7e3b01e7ec2b2" member_type: VOTER } }
I20260807 11:08:25.461318 24129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0244322652a34ae4b3b7e3b01e7ec2b2 [sys.catalog]: This master's current role is: LEADER
I20260807 11:08:25.461308 24128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0244322652a34ae4b3b7e3b01e7ec2b2 [sys.catalog]: This master's current role is: LEADER
I20260807 11:08:25.486555 24138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:08:25.514134 24138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:08:25.546694 24034 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260807 11:08:25.560828 24138 catalog_manager.cc:1347] Generated new cluster ID: db3677ba00d241d8a5bc878777bcd5e8
I20260807 11:08:25.561453 24138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:08:25.591488 24138 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:08:25.595005 24138 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:08:25.613353 24138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0244322652a34ae4b3b7e3b01e7ec2b2: Generated new TSK 0
I20260807 11:08:25.614996 24138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:08:25.626240 24034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:08:25.650959 24157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:08:25.655371 24158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:08:25.673353 24160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:08:25.675974 24034 server_base.cc:1034] running on GCE node
I20260807 11:08:25.676962 24034 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 11:08:25.677484 24034 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 11:08:25.677734 24034 hybrid_clock.cc:648] HybridClock initialized: now 1786100905677709 us; error 0 us; skew 500 ppm
I20260807 11:08:25.678412 24034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:08:25.689582 24034 webserver.cc:460] Webserver started at http://127.23.120.129:34675/ using document root <none> and password file <none>
I20260807 11:08:25.690296 24034 fs_manager.cc:359] Metadata directory not provided
I20260807 11:08:25.690542 24034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:08:25.690919 24034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:08:25.692476 24034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786100902301536-24034-0/minicluster-data/ts-0-root/instance:
uuid: "625cae382881486e90d3a22d03b7ef7a"
format_stamp: "Formatted at 2026-08-07 11:08:25 on dist-test-slave-g170"
I20260807 11:08:25.704136 24034 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.004s sys 0.005s
I20260807 11:08:25.714134 24167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:08:25.715507 24034 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.007s
I20260807 11:08:25.715837 24034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786100902301536-24034-0/minicluster-data/ts-0-root
uuid: "625cae382881486e90d3a22d03b7ef7a"
format_stamp: "Formatted at 2026-08-07 11:08:25 on dist-test-slave-g170"
I20260807 11:08:25.716192 24034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786100902301536-24034-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786100902301536-24034-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786100902301536-24034-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:08:25.765743 24034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:08:25.768332 24034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:08:25.777781 24034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:08:25.789947 24034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:08:25.790297 24034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:08:25.790617 24034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:08:25.790773 24034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:08:26.144181 24034 rpc_server.cc:307] RPC server started. Bound to: 127.23.120.129:38841
I20260807 11:08:26.144657 24261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.120.129:38841 every 8 connection(s)
I20260807 11:08:26.162281 24034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:08:26.196385 24271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:08:26.203526 24273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:08:26.225018 24034 server_base.cc:1034] running on GCE node
W20260807 11:08:26.227176 24275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:08:26.228451 24034 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 11:08:26.228675 24034 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 11:08:26.228942 24034 hybrid_clock.cc:648] HybridClock initialized: now 1786100906228925 us; error 0 us; skew 500 ppm
I20260807 11:08:26.229612 24034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:08:26.232656 24034 webserver.cc:460] Webserver started at http://127.23.120.130:39715/ using document root <none> and password file <none>
I20260807 11:08:26.233201 24034 fs_manager.cc:359] Metadata directory not provided
I20260807 11:08:26.233414 24034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:08:26.233703 24034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:08:26.235265 24034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786100902301536-24034-0/minicluster-data/ts-1-root/instance:
uuid: "2e04f9d945c14636bc08ef0f3183fa79"
format_stamp: "Formatted at 2026-08-07 11:08:26 on dist-test-slave-g170"
I20260807 11:08:26.239336 24265 heartbeater.cc:347] Connected to a master server at 127.23.120.190:46829
I20260807 11:08:26.240000 24265 heartbeater.cc:464] Registering TS with master...
I20260807 11:08:26.241762 24265 heartbeater.cc:511] Master 127.23.120.190:46829 requested a full tablet report, sending...
I20260807 11:08:26.241981 24034 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260807 11:08:26.245910 24080 ts_manager.cc:194] Registered new tserver with Master: 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841)
I20260807 11:08:26.248456 24282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:08:26.249953 24034 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.001s
I20260807 11:08:26.250310 24034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786100902301536-24034-0/minicluster-data/ts-1-root
uuid: "2e04f9d945c14636bc08ef0f3183fa79"
format_stamp: "Formatted at 2026-08-07 11:08:26 on dist-test-slave-g170"
I20260807 11:08:26.250730 24034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786100902301536-24034-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786100902301536-24034-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786100902301536-24034-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:08:26.251792 24080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50432
I20260807 11:08:26.286003 24034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:08:26.288862 24034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:08:26.297611 24034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:08:26.307128 24034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:08:26.307463 24034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:08:26.307817 24034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:08:26.308147 24034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260807 11:08:26.378283 24117 debug-util.cc:398] Leaking SignalData structure 0x7b0800060060 after lost signal to thread 24038
I20260807 11:08:26.753098 24034 rpc_server.cc:307] RPC server started. Bound to: 127.23.120.130:39129
I20260807 11:08:26.753675 24369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.120.130:39129 every 8 connection(s)
I20260807 11:08:26.789636 24034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:08:26.826092 24377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:08:26.833860 24371 heartbeater.cc:347] Connected to a master server at 127.23.120.190:46829
I20260807 11:08:26.834839 24371 heartbeater.cc:464] Registering TS with master...
W20260807 11:08:26.835139 24378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:08:26.837271 24371 heartbeater.cc:511] Master 127.23.120.190:46829 requested a full tablet report, sending...
I20260807 11:08:26.843142 24080 ts_manager.cc:194] Registered new tserver with Master: 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129)
I20260807 11:08:26.848119 24080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55708
I20260807 11:08:26.876341 24034 server_base.cc:1034] running on GCE node
W20260807 11:08:26.883630 24382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:08:26.885133 24034 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 11:08:26.885366 24034 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 11:08:26.885555 24034 hybrid_clock.cc:648] HybridClock initialized: now 1786100906885538 us; error 0 us; skew 500 ppm
I20260807 11:08:26.886096 24034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:08:26.889204 24034 webserver.cc:460] Webserver started at http://127.23.120.131:36425/ using document root <none> and password file <none>
I20260807 11:08:26.889719 24034 fs_manager.cc:359] Metadata directory not provided
I20260807 11:08:26.889932 24034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:08:26.890216 24034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:08:26.891323 24034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786100902301536-24034-0/minicluster-data/ts-2-root/instance:
uuid: "212ebf83bd07438a82bd50130d0f427d"
format_stamp: "Formatted at 2026-08-07 11:08:26 on dist-test-slave-g170"
I20260807 11:08:26.896337 24034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260807 11:08:26.900404 24390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:08:26.901332 24034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:08:26.901619 24034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786100902301536-24034-0/minicluster-data/ts-2-root
uuid: "212ebf83bd07438a82bd50130d0f427d"
format_stamp: "Formatted at 2026-08-07 11:08:26 on dist-test-slave-g170"
I20260807 11:08:26.901907 24034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786100902301536-24034-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786100902301536-24034-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786100902301536-24034-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:08:26.919899 24034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:08:26.921221 24034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:08:26.926723 24034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:08:26.930374 24034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:08:26.930598 24034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:08:26.930855 24034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:08:26.931025 24034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:08:27.258276 24265 heartbeater.cc:503] Master 127.23.120.190:46829 was elected leader, sending a full tablet report...
I20260807 11:08:27.315413 24034 rpc_server.cc:307] RPC server started. Bound to: 127.23.120.131:43609
I20260807 11:08:27.316582 24477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.120.131:43609 every 8 connection(s)
I20260807 11:08:27.368861 24478 heartbeater.cc:347] Connected to a master server at 127.23.120.190:46829
I20260807 11:08:27.369433 24478 heartbeater.cc:464] Registering TS with master...
I20260807 11:08:27.370745 24478 heartbeater.cc:511] Master 127.23.120.190:46829 requested a full tablet report, sending...
I20260807 11:08:27.373751 24080 ts_manager.cc:194] Registered new tserver with Master: 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:27.373955 24034 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.033017612s
I20260807 11:08:27.377216 24080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55716
I20260807 11:08:27.386901 24034 test_util.cc:268] Using random seed: 102639611
I20260807 11:08:27.516583 24080 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55724:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:08:27.527056 24080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:08:27.859019 24371 heartbeater.cc:503] Master 127.23.120.190:46829 was elected leader, sending a full tablet report...
I20260807 11:08:27.849521 24427 tablet_service.cc:1467] Processing CreateTablet for tablet 5a1d2a0a8e38476ea55fdd85717cf7ea (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260807 11:08:27.865384 24320 tablet_service.cc:1467] Processing CreateTablet for tablet 9f8dec87030d45c9a4b51dd3842292d1 (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260807 11:08:27.856455 24428 tablet_service.cc:1467] Processing CreateTablet for tablet 9f8dec87030d45c9a4b51dd3842292d1 (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260807 11:08:27.859390 24426 tablet_service.cc:1467] Processing CreateTablet for tablet b0993a417f1b4e32ab72ff3d30fbacef (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260807 11:08:27.865708 24424 tablet_service.cc:1467] Processing CreateTablet for tablet 143ad113a591461abce196dc159fa9f7 (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260807 11:08:27.879917 24425 tablet_service.cc:1467] Processing CreateTablet for tablet 1308b5fb7ff942a69ffe1f28a83642bb (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260807 11:08:27.889503 24427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a1d2a0a8e38476ea55fdd85717cf7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:27.897861 24320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f8dec87030d45c9a4b51dd3842292d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:27.880486 24423 tablet_service.cc:1467] Processing CreateTablet for tablet 1d0a8dfa345949b192adaddd29002763 (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260807 11:08:27.901095 24420 tablet_service.cc:1467] Processing CreateTablet for tablet 9dff94141ffa4c789f8e39fcdfe20b9b (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260807 11:08:27.913741 24428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f8dec87030d45c9a4b51dd3842292d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:27.907223 24317 tablet_service.cc:1467] Processing CreateTablet for tablet 5a1d2a0a8e38476ea55fdd85717cf7ea (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260807 11:08:27.924595 24317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a1d2a0a8e38476ea55fdd85717cf7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:27.918046 24315 tablet_service.cc:1467] Processing CreateTablet for tablet b0993a417f1b4e32ab72ff3d30fbacef (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260807 11:08:27.907369 24211 tablet_service.cc:1467] Processing CreateTablet for tablet 5a1d2a0a8e38476ea55fdd85717cf7ea (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260807 11:08:27.934836 24425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1308b5fb7ff942a69ffe1f28a83642bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:27.928308 24212 tablet_service.cc:1467] Processing CreateTablet for tablet 9f8dec87030d45c9a4b51dd3842292d1 (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260807 11:08:27.943898 24315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0993a417f1b4e32ab72ff3d30fbacef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:27.935130 24207 tablet_service.cc:1467] Processing CreateTablet for tablet 1308b5fb7ff942a69ffe1f28a83642bb (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260807 11:08:27.941665 24312 tablet_service.cc:1467] Processing CreateTablet for tablet 143ad113a591461abce196dc159fa9f7 (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260807 11:08:27.941093 24208 tablet_service.cc:1467] Processing CreateTablet for tablet b0993a417f1b4e32ab72ff3d30fbacef (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260807 11:08:27.935395 24418 tablet_service.cc:1467] Processing CreateTablet for tablet bc59e5e4c27b459e938d8942f3ff65c3 (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260807 11:08:27.945447 24423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d0a8dfa345949b192adaddd29002763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:27.937263 24419 tablet_service.cc:1467] Processing CreateTablet for tablet d9426ebc1a024747ad29dd72e9a763d4 (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260807 11:08:27.941067 24314 tablet_service.cc:1467] Processing CreateTablet for tablet 1308b5fb7ff942a69ffe1f28a83642bb (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260807 11:08:27.962004 24205 tablet_service.cc:1467] Processing CreateTablet for tablet 1d0a8dfa345949b192adaddd29002763 (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260807 11:08:27.953130 24309 tablet_service.cc:1467] Processing CreateTablet for tablet d9426ebc1a024747ad29dd72e9a763d4 (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260807 11:08:27.971019 24420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dff94141ffa4c789f8e39fcdfe20b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:27.953399 24206 tablet_service.cc:1467] Processing CreateTablet for tablet 143ad113a591461abce196dc159fa9f7 (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260807 11:08:27.968606 24312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 143ad113a591461abce196dc159fa9f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:27.963562 24311 tablet_service.cc:1467] Processing CreateTablet for tablet 1d0a8dfa345949b192adaddd29002763 (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260807 11:08:27.987979 24211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a1d2a0a8e38476ea55fdd85717cf7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:27.997334 24309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9426ebc1a024747ad29dd72e9a763d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:27.971324 24204 tablet_service.cc:1467] Processing CreateTablet for tablet 9dff94141ffa4c789f8e39fcdfe20b9b (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260807 11:08:27.969781 24310 tablet_service.cc:1467] Processing CreateTablet for tablet 9dff94141ffa4c789f8e39fcdfe20b9b (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260807 11:08:27.987267 24202 tablet_service.cc:1467] Processing CreateTablet for tablet d9426ebc1a024747ad29dd72e9a763d4 (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260807 11:08:28.003233 24208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0993a417f1b4e32ab72ff3d30fbacef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:28.004031 24418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc59e5e4c27b459e938d8942f3ff65c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:27.983009 24307 tablet_service.cc:1467] Processing CreateTablet for tablet bc59e5e4c27b459e938d8942f3ff65c3 (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260807 11:08:28.005640 24201 tablet_service.cc:1467] Processing CreateTablet for tablet bc59e5e4c27b459e938d8942f3ff65c3 (DEFAULT_TABLE table=test-workload [id=9af892c7ab714060b0213a2e21a49d5f]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260807 11:08:28.010673 24311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d0a8dfa345949b192adaddd29002763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:28.015340 24205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d0a8dfa345949b192adaddd29002763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:28.019915 24212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f8dec87030d45c9a4b51dd3842292d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:28.016630 24314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1308b5fb7ff942a69ffe1f28a83642bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:28.027319 24202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9426ebc1a024747ad29dd72e9a763d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:28.030041 24201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc59e5e4c27b459e938d8942f3ff65c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:28.034463 24207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1308b5fb7ff942a69ffe1f28a83642bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:28.042665 24307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc59e5e4c27b459e938d8942f3ff65c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:28.033663 24419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9426ebc1a024747ad29dd72e9a763d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:28.053298 24206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 143ad113a591461abce196dc159fa9f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:28.059053 24204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dff94141ffa4c789f8e39fcdfe20b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:28.053300 24310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dff94141ffa4c789f8e39fcdfe20b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:28.082496 24426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0993a417f1b4e32ab72ff3d30fbacef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:28.140323 24424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 143ad113a591461abce196dc159fa9f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:28.267681 24509 tablet_bootstrap.cc:492] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79: Bootstrap starting.
I20260807 11:08:28.382236 24510 tablet_bootstrap.cc:492] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a: Bootstrap starting.
I20260807 11:08:28.384003 24478 heartbeater.cc:503] Master 127.23.120.190:46829 was elected leader, sending a full tablet report...
I20260807 11:08:28.397696 24509 tablet_bootstrap.cc:654] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:28.475359 24510 tablet_bootstrap.cc:654] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:28.483664 24512 tablet_bootstrap.cc:492] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d: Bootstrap starting.
I20260807 11:08:28.501541 24509 tablet_bootstrap.cc:492] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79: No bootstrap required, opened a new log
I20260807 11:08:28.503763 24509 ts_tablet_manager.cc:1397] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79: Time spent bootstrapping tablet: real 0.238s user 0.009s sys 0.027s
W20260807 11:08:28.510489 24142 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 11:08:28.543031 24509 raft_consensus.cc:348] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:28.544719 24509 raft_consensus.cc:374] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:28.547293 24509 raft_consensus.cc:729] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e04f9d945c14636bc08ef0f3183fa79, State: Initialized, Role: FOLLOWER
I20260807 11:08:28.548237 24510 tablet_bootstrap.cc:492] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a: No bootstrap required, opened a new log
I20260807 11:08:28.548964 24510 ts_tablet_manager.cc:1397] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a: Time spent bootstrapping tablet: real 0.204s user 0.028s sys 0.043s
I20260807 11:08:28.547986 24512 tablet_bootstrap.cc:654] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:28.548823 24509 consensus_queue.cc:260] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:28.557250 24510 raft_consensus.cc:348] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:28.558529 24510 raft_consensus.cc:374] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:28.559070 24510 raft_consensus.cc:729] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625cae382881486e90d3a22d03b7ef7a, State: Initialized, Role: FOLLOWER
I20260807 11:08:28.561524 24509 ts_tablet_manager.cc:1428] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79: Time spent starting tablet: real 0.057s user 0.008s sys 0.014s
I20260807 11:08:28.562872 24510 consensus_queue.cc:260] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:28.567210 24509 tablet_bootstrap.cc:492] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79: Bootstrap starting.
I20260807 11:08:28.598430 24509 tablet_bootstrap.cc:654] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:28.605571 24512 tablet_bootstrap.cc:492] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d: No bootstrap required, opened a new log
I20260807 11:08:28.606288 24512 ts_tablet_manager.cc:1397] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d: Time spent bootstrapping tablet: real 0.128s user 0.020s sys 0.016s
I20260807 11:08:28.608230 24510 ts_tablet_manager.cc:1428] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a: Time spent starting tablet: real 0.059s user 0.016s sys 0.014s
I20260807 11:08:28.611536 24512 raft_consensus.cc:348] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:28.612619 24512 raft_consensus.cc:374] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:28.613046 24512 raft_consensus.cc:729] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212ebf83bd07438a82bd50130d0f427d, State: Initialized, Role: FOLLOWER
I20260807 11:08:28.614250 24512 consensus_queue.cc:260] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:28.619763 24510 tablet_bootstrap.cc:492] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a: Bootstrap starting.
I20260807 11:08:28.635365 24509 tablet_bootstrap.cc:492] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79: No bootstrap required, opened a new log
I20260807 11:08:28.636320 24509 ts_tablet_manager.cc:1397] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79: Time spent bootstrapping tablet: real 0.071s user 0.014s sys 0.035s
I20260807 11:08:28.645941 24512 ts_tablet_manager.cc:1428] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d: Time spent starting tablet: real 0.039s user 0.011s sys 0.015s
I20260807 11:08:28.648078 24509 raft_consensus.cc:348] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:28.649246 24509 raft_consensus.cc:374] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:28.649704 24509 raft_consensus.cc:729] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e04f9d945c14636bc08ef0f3183fa79, State: Initialized, Role: FOLLOWER
I20260807 11:08:28.650738 24510 tablet_bootstrap.cc:654] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:28.650941 24509 consensus_queue.cc:260] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:28.658958 24509 ts_tablet_manager.cc:1428] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79: Time spent starting tablet: real 0.022s user 0.008s sys 0.004s
I20260807 11:08:28.662683 24521 raft_consensus.cc:482] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:28.663430 24521 raft_consensus.cc:504] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:28.661000 24512 tablet_bootstrap.cc:492] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d: Bootstrap starting.
I20260807 11:08:28.674782 24509 tablet_bootstrap.cc:492] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79: Bootstrap starting.
I20260807 11:08:28.677492 24521 leader_election.cc:288] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129)
W20260807 11:08:28.686131 24267 tablet.cc:2369] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:08:28.743643 24512 tablet_bootstrap.cc:654] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:28.760879 24510 tablet_bootstrap.cc:492] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a: No bootstrap required, opened a new log
I20260807 11:08:28.761478 24510 ts_tablet_manager.cc:1397] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a: Time spent bootstrapping tablet: real 0.142s user 0.012s sys 0.033s
I20260807 11:08:28.777786 24510 raft_consensus.cc:348] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:28.778905 24510 raft_consensus.cc:374] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:28.779398 24510 raft_consensus.cc:729] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625cae382881486e90d3a22d03b7ef7a, State: Initialized, Role: FOLLOWER
I20260807 11:08:28.789494 24509 tablet_bootstrap.cc:654] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:28.785740 24510 consensus_queue.cc:260] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:28.800173 24510 ts_tablet_manager.cc:1428] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a: Time spent starting tablet: real 0.038s user 0.006s sys 0.005s
I20260807 11:08:28.802439 24512 tablet_bootstrap.cc:492] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d: No bootstrap required, opened a new log
I20260807 11:08:28.803335 24512 ts_tablet_manager.cc:1397] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d: Time spent bootstrapping tablet: real 0.143s user 0.028s sys 0.028s
I20260807 11:08:28.811924 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0a8dfa345949b192adaddd29002763" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625cae382881486e90d3a22d03b7ef7a" is_pre_election: true
W20260807 11:08:28.816246 24393 leader_election.cc:341] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:28.818313 24510 tablet_bootstrap.cc:492] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a: Bootstrap starting.
I20260807 11:08:28.826956 24512 raft_consensus.cc:348] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:28.828212 24512 raft_consensus.cc:374] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:28.828712 24512 raft_consensus.cc:729] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212ebf83bd07438a82bd50130d0f427d, State: Initialized, Role: FOLLOWER
I20260807 11:08:28.829911 24512 consensus_queue.cc:260] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:28.837514 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0a8dfa345949b192adaddd29002763" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79" is_pre_election: true
I20260807 11:08:28.839782 24337 raft_consensus.cc:2454] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212ebf83bd07438a82bd50130d0f427d in term 0.
I20260807 11:08:28.844534 24399 leader_election.cc:302] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d, 2e04f9d945c14636bc08ef0f3183fa79; no voters: 625cae382881486e90d3a22d03b7ef7a
I20260807 11:08:28.847401 24523 raft_consensus.cc:2781] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:08:28.848498 24512 ts_tablet_manager.cc:1428] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d: Time spent starting tablet: real 0.045s user 0.009s sys 0.005s
I20260807 11:08:28.850183 24512 tablet_bootstrap.cc:492] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d: Bootstrap starting.
I20260807 11:08:28.848376 24523 raft_consensus.cc:482] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:08:28.856340 24523 raft_consensus.cc:3037] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:28.864943 24510 tablet_bootstrap.cc:654] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:28.884903 24523 raft_consensus.cc:504] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:28.885958 24512 tablet_bootstrap.cc:654] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:28.887096 24509 tablet_bootstrap.cc:492] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79: No bootstrap required, opened a new log
I20260807 11:08:28.887967 24509 ts_tablet_manager.cc:1397] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79: Time spent bootstrapping tablet: real 0.214s user 0.027s sys 0.024s
I20260807 11:08:28.895504 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0a8dfa345949b192adaddd29002763" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
W20260807 11:08:28.898313 24393 leader_election.cc:341] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:28.899042 24517 raft_consensus.cc:482] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:28.899827 24517 raft_consensus.cc:504] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:28.901494 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0a8dfa345949b192adaddd29002763" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
I20260807 11:08:28.902833 24337 raft_consensus.cc:3037] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:28.900326 24523 leader_election.cc:288] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 1 election: Requested vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129)
I20260807 11:08:28.916630 24517 leader_election.cc:288] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:28.935259 24509 raft_consensus.cc:348] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:28.936683 24509 raft_consensus.cc:374] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:28.937350 24509 raft_consensus.cc:729] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e04f9d945c14636bc08ef0f3183fa79, State: Initialized, Role: FOLLOWER
I20260807 11:08:28.950485 24516 raft_consensus.cc:482] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:28.956981 24516 raft_consensus.cc:504] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:28.961761 24337 raft_consensus.cc:2454] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212ebf83bd07438a82bd50130d0f427d in term 1.
I20260807 11:08:28.975359 24510 tablet_bootstrap.cc:492] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a: No bootstrap required, opened a new log
I20260807 11:08:28.976112 24510 ts_tablet_manager.cc:1397] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a: Time spent bootstrapping tablet: real 0.158s user 0.004s sys 0.025s
I20260807 11:08:28.956384 24509 consensus_queue.cc:260] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:28.982786 24399 leader_election.cc:302] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d, 2e04f9d945c14636bc08ef0f3183fa79; no voters: 625cae382881486e90d3a22d03b7ef7a
I20260807 11:08:28.986835 24523 raft_consensus.cc:2781] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:08:28.987754 24523 raft_consensus.cc:686] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [term 1 LEADER]: Becoming Leader. State: Replica: 212ebf83bd07438a82bd50130d0f427d, State: Running, Role: LEADER
I20260807 11:08:28.968077 24516 leader_election.cc:288] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:28.989490 24523 consensus_queue.cc:237] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.013211 24517 raft_consensus.cc:482] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:29.018612 24517 raft_consensus.cc:504] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.023674 24510 raft_consensus.cc:348] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.025045 24510 raft_consensus.cc:374] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:29.025578 24510 raft_consensus.cc:729] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625cae382881486e90d3a22d03b7ef7a, State: Initialized, Role: FOLLOWER
I20260807 11:08:29.038658 24510 consensus_queue.cc:260] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.048592 24512 tablet_bootstrap.cc:492] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d: No bootstrap required, opened a new log
I20260807 11:08:29.050050 24512 ts_tablet_manager.cc:1397] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d: Time spent bootstrapping tablet: real 0.200s user 0.013s sys 0.050s
I20260807 11:08:29.046587 24517 leader_election.cc:288] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:29.062830 24510 ts_tablet_manager.cc:1428] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a: Time spent starting tablet: real 0.086s user 0.016s sys 0.004s
I20260807 11:08:29.093078 24510 tablet_bootstrap.cc:492] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a: Bootstrap starting.
I20260807 11:08:29.106204 24512 raft_consensus.cc:348] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.105662 24509 ts_tablet_manager.cc:1428] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79: Time spent starting tablet: real 0.215s user 0.022s sys 0.020s
I20260807 11:08:29.107897 24512 raft_consensus.cc:374] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:29.109885 24512 raft_consensus.cc:729] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212ebf83bd07438a82bd50130d0f427d, State: Initialized, Role: FOLLOWER
I20260807 11:08:29.116994 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1308b5fb7ff942a69ffe1f28a83642bb" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79" is_pre_election: true
I20260807 11:08:29.119012 24335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0993a417f1b4e32ab72ff3d30fbacef" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79" is_pre_election: true
W20260807 11:08:29.124791 24175 leader_election.cc:341] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:29.118767 24512 consensus_queue.cc:260] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.127436 24509 tablet_bootstrap.cc:492] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79: Bootstrap starting.
W20260807 11:08:29.140918 24175 leader_election.cc:341] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:29.142788 24080 catalog_manager.cc:5495] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d reported cstate change: term changed from 0 to 1, leader changed from <none> to 212ebf83bd07438a82bd50130d0f427d (127.23.120.131). New cstate: current_term: 1 leader_uuid: "212ebf83bd07438a82bd50130d0f427d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: HEALTHY } } }
I20260807 11:08:29.148074 24512 ts_tablet_manager.cc:1428] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d: Time spent starting tablet: real 0.091s user 0.001s sys 0.013s
I20260807 11:08:29.154254 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1308b5fb7ff942a69ffe1f28a83642bb" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212ebf83bd07438a82bd50130d0f427d" is_pre_election: true
I20260807 11:08:29.155491 24442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0993a417f1b4e32ab72ff3d30fbacef" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212ebf83bd07438a82bd50130d0f427d" is_pre_election: true
I20260807 11:08:29.157091 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1d2a0a8e38476ea55fdd85717cf7ea" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625cae382881486e90d3a22d03b7ef7a" is_pre_election: true
I20260807 11:08:29.157755 24512 tablet_bootstrap.cc:492] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d: Bootstrap starting.
W20260807 11:08:29.160482 24174 leader_election.cc:341] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609): Illegal state: must be running to vote when last-logged opid is not known
W20260807 11:08:29.163252 24283 leader_election.cc:341] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:29.161063 24174 leader_election.cc:302] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 625cae382881486e90d3a22d03b7ef7a; no voters: 212ebf83bd07438a82bd50130d0f427d, 2e04f9d945c14636bc08ef0f3183fa79
W20260807 11:08:29.169245 24174 leader_election.cc:341] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:29.166530 24544 raft_consensus.cc:2726] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:08:29.170270 24174 leader_election.cc:302] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 625cae382881486e90d3a22d03b7ef7a; no voters: 212ebf83bd07438a82bd50130d0f427d, 2e04f9d945c14636bc08ef0f3183fa79
I20260807 11:08:29.173030 24544 raft_consensus.cc:2726] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:08:29.183913 24510 tablet_bootstrap.cc:654] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:29.192027 24512 tablet_bootstrap.cc:654] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:29.202494 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1d2a0a8e38476ea55fdd85717cf7ea" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212ebf83bd07438a82bd50130d0f427d" is_pre_election: true
W20260807 11:08:29.206797 24285 leader_election.cc:341] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:29.207641 24285 leader_election.cc:302] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e04f9d945c14636bc08ef0f3183fa79; no voters: 212ebf83bd07438a82bd50130d0f427d, 625cae382881486e90d3a22d03b7ef7a
I20260807 11:08:29.210027 24543 raft_consensus.cc:2726] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:08:29.222450 24509 tablet_bootstrap.cc:654] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:29.226529 24510 tablet_bootstrap.cc:492] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a: No bootstrap required, opened a new log
I20260807 11:08:29.227195 24510 ts_tablet_manager.cc:1397] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a: Time spent bootstrapping tablet: real 0.135s user 0.014s sys 0.023s
I20260807 11:08:29.246584 24544 raft_consensus.cc:482] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:29.247610 24544 raft_consensus.cc:504] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.252564 24544 leader_election.cc:288] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:29.237020 24510 raft_consensus.cc:348] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.254375 24510 raft_consensus.cc:374] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:29.254768 24510 raft_consensus.cc:729] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625cae382881486e90d3a22d03b7ef7a, State: Initialized, Role: FOLLOWER
I20260807 11:08:29.254660 24335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc59e5e4c27b459e938d8942f3ff65c3" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79" is_pre_election: true
I20260807 11:08:29.255892 24510 consensus_queue.cc:260] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.256672 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc59e5e4c27b459e938d8942f3ff65c3" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212ebf83bd07438a82bd50130d0f427d" is_pre_election: true
I20260807 11:08:29.257700 24443 raft_consensus.cc:2454] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 625cae382881486e90d3a22d03b7ef7a in term 0.
I20260807 11:08:29.260041 24510 ts_tablet_manager.cc:1428] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a: Time spent starting tablet: real 0.032s user 0.004s sys 0.006s
I20260807 11:08:29.259850 24174 leader_election.cc:302] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d, 625cae382881486e90d3a22d03b7ef7a; no voters:
I20260807 11:08:29.261601 24543 raft_consensus.cc:482] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:08:29.261904 24175 leader_election.cc:341] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:29.262331 24543 raft_consensus.cc:504] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.268867 24543 leader_election.cc:288] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:29.269112 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8dec87030d45c9a4b51dd3842292d1" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625cae382881486e90d3a22d03b7ef7a" is_pre_election: true
I20260807 11:08:29.264151 24544 raft_consensus.cc:2781] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:08:29.270063 24544 raft_consensus.cc:482] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:08:29.270675 24544 raft_consensus.cc:3037] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:29.271129 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8dec87030d45c9a4b51dd3842292d1" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212ebf83bd07438a82bd50130d0f427d" is_pre_election: true
W20260807 11:08:29.271553 24283 leader_election.cc:341] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841): Illegal state: must be running to vote when last-logged opid is not known
W20260807 11:08:29.273630 24285 leader_election.cc:341] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:29.274482 24285 leader_election.cc:302] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e04f9d945c14636bc08ef0f3183fa79; no voters: 212ebf83bd07438a82bd50130d0f427d, 625cae382881486e90d3a22d03b7ef7a
I20260807 11:08:29.276050 24543 raft_consensus.cc:2726] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:08:29.287017 24510 tablet_bootstrap.cc:492] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a: Bootstrap starting.
I20260807 11:08:29.306102 24512 tablet_bootstrap.cc:492] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d: No bootstrap required, opened a new log
I20260807 11:08:29.308516 24512 ts_tablet_manager.cc:1397] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d: Time spent bootstrapping tablet: real 0.151s user 0.033s sys 0.036s
I20260807 11:08:29.314530 24512 raft_consensus.cc:348] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.315872 24512 raft_consensus.cc:374] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:29.321100 24509 tablet_bootstrap.cc:492] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79: No bootstrap required, opened a new log
I20260807 11:08:29.321955 24509 ts_tablet_manager.cc:1397] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79: Time spent bootstrapping tablet: real 0.202s user 0.013s sys 0.027s
I20260807 11:08:29.314956 24544 raft_consensus.cc:504] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.328855 24335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc59e5e4c27b459e938d8942f3ff65c3" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
W20260807 11:08:29.330655 24175 leader_election.cc:341] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:29.316468 24512 raft_consensus.cc:729] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212ebf83bd07438a82bd50130d0f427d, State: Initialized, Role: FOLLOWER
I20260807 11:08:29.335301 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc59e5e4c27b459e938d8942f3ff65c3" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
I20260807 11:08:29.336081 24544 leader_election.cc:288] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 election: Requested vote from peers 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:29.336171 24443 raft_consensus.cc:3037] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:29.338344 24509 raft_consensus.cc:348] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.341238 24509 raft_consensus.cc:374] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:29.341740 24509 raft_consensus.cc:729] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e04f9d945c14636bc08ef0f3183fa79, State: Initialized, Role: FOLLOWER
I20260807 11:08:29.340932 24510 tablet_bootstrap.cc:654] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:29.334019 24512 consensus_queue.cc:260] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.346005 24509 consensus_queue.cc:260] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.360257 24443 raft_consensus.cc:2454] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625cae382881486e90d3a22d03b7ef7a in term 1.
I20260807 11:08:29.362208 24174 leader_election.cc:302] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d, 625cae382881486e90d3a22d03b7ef7a; no voters: 2e04f9d945c14636bc08ef0f3183fa79
I20260807 11:08:29.364189 24512 ts_tablet_manager.cc:1428] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d: Time spent starting tablet: real 0.055s user 0.007s sys 0.003s
I20260807 11:08:29.365054 24544 raft_consensus.cc:2781] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:08:29.365768 24544 raft_consensus.cc:686] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [term 1 LEADER]: Becoming Leader. State: Replica: 625cae382881486e90d3a22d03b7ef7a, State: Running, Role: LEADER
I20260807 11:08:29.367030 24512 tablet_bootstrap.cc:492] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d: Bootstrap starting.
I20260807 11:08:29.367174 24544 consensus_queue.cc:237] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.372902 24509 ts_tablet_manager.cc:1428] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79: Time spent starting tablet: real 0.050s user 0.006s sys 0.006s
I20260807 11:08:29.374454 24509 tablet_bootstrap.cc:492] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79: Bootstrap starting.
I20260807 11:08:29.392581 24510 tablet_bootstrap.cc:492] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a: No bootstrap required, opened a new log
I20260807 11:08:29.401794 24512 tablet_bootstrap.cc:654] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:29.402441 24510 ts_tablet_manager.cc:1397] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a: Time spent bootstrapping tablet: real 0.116s user 0.007s sys 0.034s
I20260807 11:08:29.412680 24509 tablet_bootstrap.cc:654] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:29.424907 24080 catalog_manager.cc:5495] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 625cae382881486e90d3a22d03b7ef7a (127.23.120.129). New cstate: current_term: 1 leader_uuid: "625cae382881486e90d3a22d03b7ef7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: UNKNOWN } } }
I20260807 11:08:29.428691 24510 raft_consensus.cc:348] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.431708 24510 raft_consensus.cc:374] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:29.432220 24512 tablet_bootstrap.cc:492] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d: No bootstrap required, opened a new log
I20260807 11:08:29.432175 24510 raft_consensus.cc:729] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625cae382881486e90d3a22d03b7ef7a, State: Initialized, Role: FOLLOWER
I20260807 11:08:29.433030 24512 ts_tablet_manager.cc:1397] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d: Time spent bootstrapping tablet: real 0.066s user 0.011s sys 0.036s
I20260807 11:08:29.442095 24512 raft_consensus.cc:348] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.443321 24512 raft_consensus.cc:374] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:29.443825 24512 raft_consensus.cc:729] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212ebf83bd07438a82bd50130d0f427d, State: Initialized, Role: FOLLOWER
I20260807 11:08:29.433565 24510 consensus_queue.cc:260] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.453796 24512 consensus_queue.cc:260] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.460573 24510 ts_tablet_manager.cc:1428] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a: Time spent starting tablet: real 0.058s user 0.003s sys 0.010s
I20260807 11:08:29.463747 24509 tablet_bootstrap.cc:492] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79: No bootstrap required, opened a new log
I20260807 11:08:29.465571 24509 ts_tablet_manager.cc:1397] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79: Time spent bootstrapping tablet: real 0.091s user 0.007s sys 0.019s
I20260807 11:08:29.468973 24544 raft_consensus.cc:482] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:29.469713 24544 raft_consensus.cc:504] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.462724 24512 ts_tablet_manager.cc:1428] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d: Time spent starting tablet: real 0.029s user 0.008s sys 0.006s
I20260807 11:08:29.473575 24512 tablet_bootstrap.cc:492] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 212ebf83bd07438a82bd50130d0f427d: Bootstrap starting.
I20260807 11:08:29.475801 24335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1d2a0a8e38476ea55fdd85717cf7ea" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79" is_pre_election: true
I20260807 11:08:29.476840 24335 raft_consensus.cc:2454] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 625cae382881486e90d3a22d03b7ef7a in term 0.
I20260807 11:08:29.477530 24510 tablet_bootstrap.cc:492] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a: Bootstrap starting.
I20260807 11:08:29.471807 24509 raft_consensus.cc:348] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.481312 24509 raft_consensus.cc:374] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:29.481817 24509 raft_consensus.cc:729] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e04f9d945c14636bc08ef0f3183fa79, State: Initialized, Role: FOLLOWER
I20260807 11:08:29.481318 24175 leader_election.cc:302] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e04f9d945c14636bc08ef0f3183fa79, 625cae382881486e90d3a22d03b7ef7a; no voters:
W20260807 11:08:29.483407 24393 consensus_peers.cc:487] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d -> Peer 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841): Couldn't send request to peer 625cae382881486e90d3a22d03b7ef7a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:08:29.486384 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1d2a0a8e38476ea55fdd85717cf7ea" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212ebf83bd07438a82bd50130d0f427d" is_pre_election: true
I20260807 11:08:29.482930 24509 consensus_queue.cc:260] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
W20260807 11:08:29.489331 24174 leader_election.cc:341] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:29.495241 24517 raft_consensus.cc:2781] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:08:29.495985 24517 raft_consensus.cc:482] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:08:29.496645 24517 raft_consensus.cc:3037] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:29.494923 24544 leader_election.cc:288] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:29.500998 24512 tablet_bootstrap.cc:654] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 212ebf83bd07438a82bd50130d0f427d: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:29.508208 24509 ts_tablet_manager.cc:1428] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79: Time spent starting tablet: real 0.042s user 0.000s sys 0.012s
I20260807 11:08:29.516487 24517 raft_consensus.cc:504] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
W20260807 11:08:29.519084 24142 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 11:08:29.520982 24335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1d2a0a8e38476ea55fdd85717cf7ea" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
I20260807 11:08:29.521982 24335 raft_consensus.cc:3037] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:29.524271 24509 tablet_bootstrap.cc:492] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79: Bootstrap starting.
I20260807 11:08:29.528602 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1d2a0a8e38476ea55fdd85717cf7ea" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
W20260807 11:08:29.530787 24174 leader_election.cc:341] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:29.534253 24517 leader_election.cc:288] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 election: Requested vote from peers 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:29.535581 24523 raft_consensus.cc:482] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:29.536238 24335 raft_consensus.cc:2454] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625cae382881486e90d3a22d03b7ef7a in term 1.
I20260807 11:08:29.536214 24523 raft_consensus.cc:504] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.538041 24175 leader_election.cc:302] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e04f9d945c14636bc08ef0f3183fa79, 625cae382881486e90d3a22d03b7ef7a; no voters: 212ebf83bd07438a82bd50130d0f427d
I20260807 11:08:29.539696 24510 tablet_bootstrap.cc:654] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:29.540340 24517 raft_consensus.cc:2781] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:08:29.542377 24517 raft_consensus.cc:686] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [term 1 LEADER]: Becoming Leader. State: Replica: 625cae382881486e90d3a22d03b7ef7a, State: Running, Role: LEADER
I20260807 11:08:29.542459 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dff94141ffa4c789f8e39fcdfe20b9b" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79" is_pre_election: true
I20260807 11:08:29.540701 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dff94141ffa4c789f8e39fcdfe20b9b" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625cae382881486e90d3a22d03b7ef7a" is_pre_election: true
I20260807 11:08:29.543907 24517 consensus_queue.cc:237] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.544308 24523 leader_election.cc:288] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129)
W20260807 11:08:29.545069 24399 leader_election.cc:341] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:29.564721 24512 tablet_bootstrap.cc:492] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 212ebf83bd07438a82bd50130d0f427d: No bootstrap required, opened a new log
W20260807 11:08:29.565485 24393 leader_election.cc:341] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:29.565891 24512 ts_tablet_manager.cc:1397] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 212ebf83bd07438a82bd50130d0f427d: Time spent bootstrapping tablet: real 0.092s user 0.008s sys 0.022s
I20260807 11:08:29.566159 24393 leader_election.cc:302] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d; no voters: 2e04f9d945c14636bc08ef0f3183fa79, 625cae382881486e90d3a22d03b7ef7a
I20260807 11:08:29.567426 24523 raft_consensus.cc:2726] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:08:29.572499 24512 raft_consensus.cc:348] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.573531 24523 consensus_queue.cc:1035] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:08:29.574927 24509 tablet_bootstrap.cc:654] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:29.575811 24544 raft_consensus.cc:482] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:29.576560 24544 raft_consensus.cc:504] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.581140 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143ad113a591461abce196dc159fa9f7" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79" is_pre_election: true
I20260807 11:08:29.581996 24337 raft_consensus.cc:2454] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 625cae382881486e90d3a22d03b7ef7a in term 0.
I20260807 11:08:29.582892 24544 leader_election.cc:288] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:29.581997 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143ad113a591461abce196dc159fa9f7" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212ebf83bd07438a82bd50130d0f427d" is_pre_election: true
I20260807 11:08:29.583645 24175 leader_election.cc:302] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e04f9d945c14636bc08ef0f3183fa79, 625cae382881486e90d3a22d03b7ef7a; no voters:
I20260807 11:08:29.584997 24544 raft_consensus.cc:2781] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260807 11:08:29.585237 24174 leader_election.cc:341] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:29.585688 24544 raft_consensus.cc:482] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:08:29.586694 24544 raft_consensus.cc:3037] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:29.588042 24512 raft_consensus.cc:374] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:29.589598 24512 raft_consensus.cc:729] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212ebf83bd07438a82bd50130d0f427d, State: Initialized, Role: FOLLOWER
I20260807 11:08:29.591081 24512 consensus_queue.cc:260] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 212ebf83bd07438a82bd50130d0f427d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.605432 24512 ts_tablet_manager.cc:1428] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 212ebf83bd07438a82bd50130d0f427d: Time spent starting tablet: real 0.039s user 0.006s sys 0.008s
I20260807 11:08:29.608031 24512 tablet_bootstrap.cc:492] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d: Bootstrap starting.
I20260807 11:08:29.619844 24510 tablet_bootstrap.cc:492] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a: No bootstrap required, opened a new log
I20260807 11:08:29.620460 24510 ts_tablet_manager.cc:1397] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a: Time spent bootstrapping tablet: real 0.143s user 0.007s sys 0.032s
I20260807 11:08:29.629662 24544 raft_consensus.cc:504] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.632125 24543 raft_consensus.cc:482] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:29.632567 24543 raft_consensus.cc:504] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.624492 24510 raft_consensus.cc:348] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.633576 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143ad113a591461abce196dc159fa9f7" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
I20260807 11:08:29.633782 24510 raft_consensus.cc:374] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:29.634254 24337 raft_consensus.cc:3037] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:29.634387 24510 raft_consensus.cc:729] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625cae382881486e90d3a22d03b7ef7a, State: Initialized, Role: FOLLOWER
I20260807 11:08:29.635526 24543 leader_election.cc:288] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:29.643038 24510 consensus_queue.cc:260] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.644304 24442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143ad113a591461abce196dc159fa9f7" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
I20260807 11:08:29.647434 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0993a417f1b4e32ab72ff3d30fbacef" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625cae382881486e90d3a22d03b7ef7a" is_pre_election: true
I20260807 11:08:29.647819 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0993a417f1b4e32ab72ff3d30fbacef" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212ebf83bd07438a82bd50130d0f427d" is_pre_election: true
I20260807 11:08:29.648568 24228 raft_consensus.cc:2454] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e04f9d945c14636bc08ef0f3183fa79 in term 0.
I20260807 11:08:29.648936 24443 raft_consensus.cc:2454] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e04f9d945c14636bc08ef0f3183fa79 in term 0.
W20260807 11:08:29.650789 24174 leader_election.cc:341] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:29.648919 24510 ts_tablet_manager.cc:1428] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a: Time spent starting tablet: real 0.028s user 0.001s sys 0.011s
I20260807 11:08:29.653146 24283 leader_election.cc:302] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e04f9d945c14636bc08ef0f3183fa79, 625cae382881486e90d3a22d03b7ef7a; no voters:
I20260807 11:08:29.650364 24544 leader_election.cc:288] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 election: Requested vote from peers 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:29.656648 24337 raft_consensus.cc:2454] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625cae382881486e90d3a22d03b7ef7a in term 1.
I20260807 11:08:29.657703 24543 raft_consensus.cc:2781] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:08:29.657862 24510 tablet_bootstrap.cc:492] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a: Bootstrap starting.
I20260807 11:08:29.658146 24543 raft_consensus.cc:482] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:08:29.659153 24543 raft_consensus.cc:3037] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:29.661679 24175 leader_election.cc:302] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e04f9d945c14636bc08ef0f3183fa79, 625cae382881486e90d3a22d03b7ef7a; no voters: 212ebf83bd07438a82bd50130d0f427d
I20260807 11:08:29.663676 24517 raft_consensus.cc:2781] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:08:29.646366 24078 catalog_manager.cc:5495] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 625cae382881486e90d3a22d03b7ef7a (127.23.120.129). New cstate: current_term: 1 leader_uuid: "625cae382881486e90d3a22d03b7ef7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: UNKNOWN } } }
I20260807 11:08:29.673568 24509 tablet_bootstrap.cc:492] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79: No bootstrap required, opened a new log
I20260807 11:08:29.674571 24509 ts_tablet_manager.cc:1397] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79: Time spent bootstrapping tablet: real 0.165s user 0.013s sys 0.021s
I20260807 11:08:29.674721 24512 tablet_bootstrap.cc:654] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:29.687616 24509 raft_consensus.cc:348] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.690637 24509 raft_consensus.cc:374] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:29.695374 24543 raft_consensus.cc:504] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.699721 24543 leader_election.cc:288] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 1 election: Requested vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:29.694267 24509 raft_consensus.cc:729] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e04f9d945c14636bc08ef0f3183fa79, State: Initialized, Role: FOLLOWER
I20260807 11:08:29.701449 24510 tablet_bootstrap.cc:654] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:29.702109 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0993a417f1b4e32ab72ff3d30fbacef" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
I20260807 11:08:29.703042 24228 raft_consensus.cc:3037] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:29.705440 24509 consensus_queue.cc:260] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.716081 24509 ts_tablet_manager.cc:1428] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79: Time spent starting tablet: real 0.040s user 0.010s sys 0.000s
I20260807 11:08:29.682616 24517 raft_consensus.cc:686] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [term 1 LEADER]: Becoming Leader. State: Replica: 625cae382881486e90d3a22d03b7ef7a, State: Running, Role: LEADER
I20260807 11:08:29.720155 24523 raft_consensus.cc:482] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:29.728348 24517 consensus_queue.cc:237] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.737699 24575 raft_consensus.cc:482] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:29.739249 24509 tablet_bootstrap.cc:492] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79: Bootstrap starting.
I20260807 11:08:29.738729 24575 raft_consensus.cc:504] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.745995 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8dec87030d45c9a4b51dd3842292d1" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79" is_pre_election: true
I20260807 11:08:29.720785 24523 raft_consensus.cc:504] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.746954 24337 raft_consensus.cc:2454] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212ebf83bd07438a82bd50130d0f427d in term 0.
I20260807 11:08:29.749257 24399 leader_election.cc:302] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d, 2e04f9d945c14636bc08ef0f3183fa79; no voters:
I20260807 11:08:29.750363 24523 leader_election.cc:288] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129)
I20260807 11:08:29.747262 24575 leader_election.cc:288] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129), 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841)
I20260807 11:08:29.777000 24510 tablet_bootstrap.cc:492] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a: No bootstrap required, opened a new log
I20260807 11:08:29.777333 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1d2a0a8e38476ea55fdd85717cf7ea" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79" is_pre_election: true
I20260807 11:08:29.782159 24337 raft_consensus.cc:2379] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212ebf83bd07438a82bd50130d0f427d in current term 1: Already voted for candidate 625cae382881486e90d3a22d03b7ef7a in this term.
I20260807 11:08:29.791052 24228 raft_consensus.cc:2454] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e04f9d945c14636bc08ef0f3183fa79 in term 1.
I20260807 11:08:29.766639 24523 raft_consensus.cc:2781] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:08:29.793972 24523 raft_consensus.cc:482] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:08:29.798195 24510 ts_tablet_manager.cc:1397] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a: Time spent bootstrapping tablet: real 0.141s user 0.019s sys 0.057s
I20260807 11:08:29.794898 24223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8dec87030d45c9a4b51dd3842292d1" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625cae382881486e90d3a22d03b7ef7a" is_pre_election: true
I20260807 11:08:29.798940 24223 raft_consensus.cc:2454] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212ebf83bd07438a82bd50130d0f427d in term 0.
I20260807 11:08:29.798097 24283 leader_election.cc:302] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e04f9d945c14636bc08ef0f3183fa79, 625cae382881486e90d3a22d03b7ef7a; no voters:
I20260807 11:08:29.801424 24543 raft_consensus.cc:2781] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:08:29.806535 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0993a417f1b4e32ab72ff3d30fbacef" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
I20260807 11:08:29.807327 24443 raft_consensus.cc:3037] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:29.808952 24543 raft_consensus.cc:686] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 LEADER]: Becoming Leader. State: Replica: 2e04f9d945c14636bc08ef0f3183fa79, State: Running, Role: LEADER
I20260807 11:08:29.799274 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1d2a0a8e38476ea55fdd85717cf7ea" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625cae382881486e90d3a22d03b7ef7a" is_pre_election: true
I20260807 11:08:29.795539 24523 raft_consensus.cc:3037] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:29.814141 24393 leader_election.cc:302] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d; no voters: 2e04f9d945c14636bc08ef0f3183fa79, 625cae382881486e90d3a22d03b7ef7a
I20260807 11:08:29.817386 24510 raft_consensus.cc:348] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.818500 24510 raft_consensus.cc:374] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:29.818935 24510 raft_consensus.cc:729] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625cae382881486e90d3a22d03b7ef7a, State: Initialized, Role: FOLLOWER
W20260807 11:08:29.821319 24175 consensus_peers.cc:487] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a -> Peer 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129): Couldn't send request to peer 2e04f9d945c14636bc08ef0f3183fa79. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:08:29.810787 24543 consensus_queue.cc:237] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.823616 24578 raft_consensus.cc:3037] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:29.822362 24510 consensus_queue.cc:260] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.831589 24512 tablet_bootstrap.cc:492] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d: No bootstrap required, opened a new log
I20260807 11:08:29.832291 24512 ts_tablet_manager.cc:1397] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d: Time spent bootstrapping tablet: real 0.224s user 0.010s sys 0.043s
I20260807 11:08:29.837960 24510 ts_tablet_manager.cc:1428] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a: Time spent starting tablet: real 0.039s user 0.008s sys 0.003s
I20260807 11:08:29.839391 24510 tablet_bootstrap.cc:492] T 9dff94141ffa4c789f8e39fcdfe20b9b P 625cae382881486e90d3a22d03b7ef7a: Bootstrap starting.
I20260807 11:08:29.839709 24509 tablet_bootstrap.cc:654] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:29.844990 24443 raft_consensus.cc:2454] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e04f9d945c14636bc08ef0f3183fa79 in term 1.
I20260807 11:08:29.855275 24523 raft_consensus.cc:504] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.860942 24335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8dec87030d45c9a4b51dd3842292d1" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
I20260807 11:08:29.861928 24335 raft_consensus.cc:3037] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:29.862700 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8dec87030d45c9a4b51dd3842292d1" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
I20260807 11:08:29.863750 24228 raft_consensus.cc:3037] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:29.868117 24523 leader_election.cc:288] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 1 election: Requested vote from peers 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129), 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841)
I20260807 11:08:29.844873 24512 raft_consensus.cc:348] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.890861 24517 consensus_queue.cc:1035] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:08:29.890152 24335 raft_consensus.cc:2454] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212ebf83bd07438a82bd50130d0f427d in term 1.
I20260807 11:08:29.894973 24399 leader_election.cc:302] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d, 2e04f9d945c14636bc08ef0f3183fa79; no voters:
I20260807 11:08:29.873518 24512 raft_consensus.cc:374] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:29.895886 24228 raft_consensus.cc:2454] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212ebf83bd07438a82bd50130d0f427d in term 1.
I20260807 11:08:29.897795 24585 raft_consensus.cc:2781] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:08:29.898823 24585 raft_consensus.cc:686] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [term 1 LEADER]: Becoming Leader. State: Replica: 212ebf83bd07438a82bd50130d0f427d, State: Running, Role: LEADER
I20260807 11:08:29.901757 24578 raft_consensus.cc:2726] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 212ebf83bd07438a82bd50130d0f427d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:08:29.895869 24512 raft_consensus.cc:729] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212ebf83bd07438a82bd50130d0f427d, State: Initialized, Role: FOLLOWER
I20260807 11:08:29.902499 24585 consensus_queue.cc:237] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.921449 24512 consensus_queue.cc:260] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:29.929966 24078 catalog_manager.cc:5495] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130). New cstate: current_term: 1 leader_uuid: "2e04f9d945c14636bc08ef0f3183fa79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: UNKNOWN } } }
I20260807 11:08:29.929730 24077 catalog_manager.cc:5495] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 625cae382881486e90d3a22d03b7ef7a (127.23.120.129). New cstate: current_term: 1 leader_uuid: "625cae382881486e90d3a22d03b7ef7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: UNKNOWN } } }
I20260807 11:08:29.931864 24512 ts_tablet_manager.cc:1428] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d: Time spent starting tablet: real 0.099s user 0.015s sys 0.000s
I20260807 11:08:29.938303 24512 tablet_bootstrap.cc:492] T d9426ebc1a024747ad29dd72e9a763d4 P 212ebf83bd07438a82bd50130d0f427d: Bootstrap starting.
I20260807 11:08:29.951875 24510 tablet_bootstrap.cc:654] T 9dff94141ffa4c789f8e39fcdfe20b9b P 625cae382881486e90d3a22d03b7ef7a: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:29.962657 24574 consensus_queue.cc:1035] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260807 11:08:29.994369 24574 consensus_queue.cc:1035] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:08:30.022049 24517 raft_consensus.cc:482] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:30.022917 24517 raft_consensus.cc:504] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:30.028637 24517 leader_election.cc:288] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:30.029347 24439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9426ebc1a024747ad29dd72e9a763d4" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212ebf83bd07438a82bd50130d0f427d" is_pre_election: true
I20260807 11:08:30.031682 24335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9426ebc1a024747ad29dd72e9a763d4" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79" is_pre_election: true
I20260807 11:08:30.032665 24335 raft_consensus.cc:2454] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 625cae382881486e90d3a22d03b7ef7a in term 0.
W20260807 11:08:30.034842 24174 leader_election.cc:341] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:30.036486 24175 leader_election.cc:302] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e04f9d945c14636bc08ef0f3183fa79, 625cae382881486e90d3a22d03b7ef7a; no voters: 212ebf83bd07438a82bd50130d0f427d
I20260807 11:08:30.040277 24517 raft_consensus.cc:2781] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:08:30.041321 24517 raft_consensus.cc:482] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:08:30.041886 24517 raft_consensus.cc:3037] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:30.057041 24509 tablet_bootstrap.cc:492] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79: No bootstrap required, opened a new log
I20260807 11:08:30.057881 24509 ts_tablet_manager.cc:1397] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79: Time spent bootstrapping tablet: real 0.319s user 0.031s sys 0.026s
I20260807 11:08:30.059301 24512 tablet_bootstrap.cc:654] T d9426ebc1a024747ad29dd72e9a763d4 P 212ebf83bd07438a82bd50130d0f427d: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:30.071131 24078 catalog_manager.cc:5495] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d reported cstate change: term changed from 0 to 1, leader changed from <none> to 212ebf83bd07438a82bd50130d0f427d (127.23.120.131). New cstate: current_term: 1 leader_uuid: "212ebf83bd07438a82bd50130d0f427d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: HEALTHY } } }
I20260807 11:08:30.074323 24517 raft_consensus.cc:504] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:30.072628 24509 raft_consensus.cc:348] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:30.103971 24509 raft_consensus.cc:374] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:30.110853 24509 raft_consensus.cc:729] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e04f9d945c14636bc08ef0f3183fa79, State: Initialized, Role: FOLLOWER
I20260807 11:08:30.112452 24509 consensus_queue.cc:260] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:30.115723 24543 raft_consensus.cc:482] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:30.116673 24543 raft_consensus.cc:504] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:30.125205 24543 leader_election.cc:288] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:30.128451 24509 ts_tablet_manager.cc:1428] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79: Time spent starting tablet: real 0.070s user 0.008s sys 0.005s
I20260807 11:08:30.131947 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dff94141ffa4c789f8e39fcdfe20b9b" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212ebf83bd07438a82bd50130d0f427d" is_pre_election: true
I20260807 11:08:30.133292 24443 raft_consensus.cc:2454] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e04f9d945c14636bc08ef0f3183fa79 in term 0.
I20260807 11:08:30.133842 24510 tablet_bootstrap.cc:492] T 9dff94141ffa4c789f8e39fcdfe20b9b P 625cae382881486e90d3a22d03b7ef7a: No bootstrap required, opened a new log
I20260807 11:08:30.134557 24510 ts_tablet_manager.cc:1397] T 9dff94141ffa4c789f8e39fcdfe20b9b P 625cae382881486e90d3a22d03b7ef7a: Time spent bootstrapping tablet: real 0.296s user 0.006s sys 0.030s
I20260807 11:08:30.135453 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dff94141ffa4c789f8e39fcdfe20b9b" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625cae382881486e90d3a22d03b7ef7a" is_pre_election: true
I20260807 11:08:30.138026 24285 leader_election.cc:302] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d, 2e04f9d945c14636bc08ef0f3183fa79; no voters:
I20260807 11:08:30.145745 24509 tablet_bootstrap.cc:492] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79: Bootstrap starting.
I20260807 11:08:30.141467 24543 raft_consensus.cc:2781] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:08:30.149633 24543 raft_consensus.cc:482] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:08:30.150178 24543 raft_consensus.cc:3037] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:30.163775 24335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9426ebc1a024747ad29dd72e9a763d4" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
W20260807 11:08:30.176254 24283 leader_election.cc:341] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:30.164541 24335 raft_consensus.cc:3037] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:30.189198 24442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9426ebc1a024747ad29dd72e9a763d4" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
I20260807 11:08:30.191792 24543 raft_consensus.cc:504] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
W20260807 11:08:30.192883 24174 leader_election.cc:341] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:30.193962 24512 tablet_bootstrap.cc:492] T d9426ebc1a024747ad29dd72e9a763d4 P 212ebf83bd07438a82bd50130d0f427d: No bootstrap required, opened a new log
I20260807 11:08:30.194936 24512 ts_tablet_manager.cc:1397] T d9426ebc1a024747ad29dd72e9a763d4 P 212ebf83bd07438a82bd50130d0f427d: Time spent bootstrapping tablet: real 0.257s user 0.019s sys 0.041s
I20260807 11:08:30.195277 24543 leader_election.cc:288] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 1 election: Requested vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:30.185601 24517 leader_election.cc:288] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 election: Requested vote from peers 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:30.201543 24510 raft_consensus.cc:348] T 9dff94141ffa4c789f8e39fcdfe20b9b P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:30.202114 24510 raft_consensus.cc:374] T 9dff94141ffa4c789f8e39fcdfe20b9b P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:30.202376 24510 raft_consensus.cc:729] T 9dff94141ffa4c789f8e39fcdfe20b9b P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625cae382881486e90d3a22d03b7ef7a, State: Initialized, Role: FOLLOWER
I20260807 11:08:30.202961 24510 consensus_queue.cc:260] T 9dff94141ffa4c789f8e39fcdfe20b9b P 625cae382881486e90d3a22d03b7ef7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:30.205019 24510 ts_tablet_manager.cc:1428] T 9dff94141ffa4c789f8e39fcdfe20b9b P 625cae382881486e90d3a22d03b7ef7a: Time spent starting tablet: real 0.070s user 0.006s sys 0.005s
I20260807 11:08:30.205852 24510 tablet_bootstrap.cc:492] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a: Bootstrap starting.
I20260807 11:08:30.205977 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dff94141ffa4c789f8e39fcdfe20b9b" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
I20260807 11:08:30.206887 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dff94141ffa4c789f8e39fcdfe20b9b" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
I20260807 11:08:30.207898 24228 raft_consensus.cc:3037] T 9dff94141ffa4c789f8e39fcdfe20b9b P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:30.206997 24443 raft_consensus.cc:3037] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:30.209998 24509 tablet_bootstrap.cc:654] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:30.232511 24228 raft_consensus.cc:2454] T 9dff94141ffa4c789f8e39fcdfe20b9b P 625cae382881486e90d3a22d03b7ef7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e04f9d945c14636bc08ef0f3183fa79 in term 1.
I20260807 11:08:30.237381 24335 raft_consensus.cc:2454] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625cae382881486e90d3a22d03b7ef7a in term 1.
I20260807 11:08:30.237596 24283 leader_election.cc:302] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e04f9d945c14636bc08ef0f3183fa79, 625cae382881486e90d3a22d03b7ef7a; no voters:
I20260807 11:08:30.238960 24543 raft_consensus.cc:2781] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:08:30.239738 24543 raft_consensus.cc:686] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 LEADER]: Becoming Leader. State: Replica: 2e04f9d945c14636bc08ef0f3183fa79, State: Running, Role: LEADER
I20260807 11:08:30.239713 24175 leader_election.cc:302] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e04f9d945c14636bc08ef0f3183fa79, 625cae382881486e90d3a22d03b7ef7a; no voters: 212ebf83bd07438a82bd50130d0f427d
I20260807 11:08:30.242281 24605 raft_consensus.cc:2781] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:08:30.241302 24543 consensus_queue.cc:237] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:30.243085 24605 raft_consensus.cc:686] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [term 1 LEADER]: Becoming Leader. State: Replica: 625cae382881486e90d3a22d03b7ef7a, State: Running, Role: LEADER
I20260807 11:08:30.239713 24512 raft_consensus.cc:348] T d9426ebc1a024747ad29dd72e9a763d4 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:30.245811 24512 raft_consensus.cc:374] T d9426ebc1a024747ad29dd72e9a763d4 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:30.246148 24512 raft_consensus.cc:729] T d9426ebc1a024747ad29dd72e9a763d4 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212ebf83bd07438a82bd50130d0f427d, State: Initialized, Role: FOLLOWER
I20260807 11:08:30.246325 24605 consensus_queue.cc:237] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:30.247077 24512 consensus_queue.cc:260] T d9426ebc1a024747ad29dd72e9a763d4 P 212ebf83bd07438a82bd50130d0f427d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:30.254725 24510 tablet_bootstrap.cc:654] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:30.256404 24512 ts_tablet_manager.cc:1428] T d9426ebc1a024747ad29dd72e9a763d4 P 212ebf83bd07438a82bd50130d0f427d: Time spent starting tablet: real 0.061s user 0.009s sys 0.001s
I20260807 11:08:30.256353 24443 raft_consensus.cc:2454] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e04f9d945c14636bc08ef0f3183fa79 in term 1.
I20260807 11:08:30.257679 24509 tablet_bootstrap.cc:492] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79: No bootstrap required, opened a new log
I20260807 11:08:30.258397 24509 ts_tablet_manager.cc:1397] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79: Time spent bootstrapping tablet: real 0.118s user 0.022s sys 0.029s
I20260807 11:08:30.265416 24509 raft_consensus.cc:348] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:30.266163 24509 raft_consensus.cc:374] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:30.266423 24509 raft_consensus.cc:729] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e04f9d945c14636bc08ef0f3183fa79, State: Initialized, Role: FOLLOWER
I20260807 11:08:30.277895 24509 consensus_queue.cc:260] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:30.289700 24509 ts_tablet_manager.cc:1428] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79: Time spent starting tablet: real 0.031s user 0.001s sys 0.010s
I20260807 11:08:30.290436 24578 raft_consensus.cc:482] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:30.292945 24509 tablet_bootstrap.cc:492] T bc59e5e4c27b459e938d8942f3ff65c3 P 2e04f9d945c14636bc08ef0f3183fa79: Bootstrap starting.
I20260807 11:08:30.292673 24578 raft_consensus.cc:504] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:30.324637 24512 tablet_bootstrap.cc:492] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d: Bootstrap starting.
I20260807 11:08:30.289973 24077 catalog_manager.cc:5495] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130). New cstate: current_term: 1 leader_uuid: "2e04f9d945c14636bc08ef0f3183fa79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: UNKNOWN } } }
I20260807 11:08:30.345466 24543 consensus_queue.cc:1035] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:08:30.349437 24578 leader_election.cc:288] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129)
I20260807 11:08:30.355455 24335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1308b5fb7ff942a69ffe1f28a83642bb" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79" is_pre_election: true
I20260807 11:08:30.356302 24335 raft_consensus.cc:2454] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212ebf83bd07438a82bd50130d0f427d in term 0.
I20260807 11:08:30.358353 24399 leader_election.cc:302] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d, 2e04f9d945c14636bc08ef0f3183fa79; no voters:
I20260807 11:08:30.360524 24578 raft_consensus.cc:2781] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:08:30.362346 24578 raft_consensus.cc:482] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:08:30.362766 24578 raft_consensus.cc:3037] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:30.371305 24517 consensus_queue.cc:1035] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:08:30.374174 24223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1308b5fb7ff942a69ffe1f28a83642bb" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625cae382881486e90d3a22d03b7ef7a" is_pre_election: true
W20260807 11:08:30.374986 24174 consensus_peers.cc:487] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a -> Peer 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609): Couldn't send request to peer 212ebf83bd07438a82bd50130d0f427d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:08:30.379812 24585 raft_consensus.cc:482] T d9426ebc1a024747ad29dd72e9a763d4 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:30.382758 24585 raft_consensus.cc:504] T d9426ebc1a024747ad29dd72e9a763d4 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:30.375084 24223 raft_consensus.cc:2454] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212ebf83bd07438a82bd50130d0f427d in term 0.
I20260807 11:08:30.394589 24523 consensus_queue.cc:1035] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [LEADER]: Connected to new peer: Peer: permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:08:30.389303 24077 catalog_manager.cc:5495] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 625cae382881486e90d3a22d03b7ef7a (127.23.120.129). New cstate: current_term: 1 leader_uuid: "625cae382881486e90d3a22d03b7ef7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: UNKNOWN } } }
I20260807 11:08:30.398797 24585 leader_election.cc:288] T d9426ebc1a024747ad29dd72e9a763d4 P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129)
I20260807 11:08:30.399335 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9426ebc1a024747ad29dd72e9a763d4" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79" is_pre_election: true
I20260807 11:08:30.399811 24543 consensus_queue.cc:1035] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:08:30.400667 24337 raft_consensus.cc:2379] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212ebf83bd07438a82bd50130d0f427d in current term 1: Already voted for candidate 625cae382881486e90d3a22d03b7ef7a in this term.
I20260807 11:08:30.439885 24512 tablet_bootstrap.cc:654] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:30.407827 24578 raft_consensus.cc:504] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:30.446305 24223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9426ebc1a024747ad29dd72e9a763d4" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625cae382881486e90d3a22d03b7ef7a" is_pre_election: true
I20260807 11:08:30.447206 24221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1308b5fb7ff942a69ffe1f28a83642bb" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
I20260807 11:08:30.449020 24221 raft_consensus.cc:3037] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:30.450018 24510 tablet_bootstrap.cc:492] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a: No bootstrap required, opened a new log
I20260807 11:08:30.450513 24510 ts_tablet_manager.cc:1397] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a: Time spent bootstrapping tablet: real 0.245s user 0.017s sys 0.022s
I20260807 11:08:30.469256 24509 tablet_bootstrap.cc:654] T bc59e5e4c27b459e938d8942f3ff65c3 P 2e04f9d945c14636bc08ef0f3183fa79: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:30.463022 24510 raft_consensus.cc:348] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:30.475723 24510 raft_consensus.cc:374] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:30.476303 24510 raft_consensus.cc:729] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625cae382881486e90d3a22d03b7ef7a, State: Initialized, Role: FOLLOWER
I20260807 11:08:30.477414 24393 leader_election.cc:302] T d9426ebc1a024747ad29dd72e9a763d4 P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d; no voters: 2e04f9d945c14636bc08ef0f3183fa79, 625cae382881486e90d3a22d03b7ef7a
I20260807 11:08:30.486377 24523 raft_consensus.cc:3037] T d9426ebc1a024747ad29dd72e9a763d4 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:30.488417 24578 leader_election.cc:288] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 1 election: Requested vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129)
I20260807 11:08:30.493462 24510 consensus_queue.cc:260] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:30.496052 24510 ts_tablet_manager.cc:1428] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a: Time spent starting tablet: real 0.045s user 0.004s sys 0.007s
I20260807 11:08:30.516714 24523 raft_consensus.cc:2726] T d9426ebc1a024747ad29dd72e9a763d4 P 212ebf83bd07438a82bd50130d0f427d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:08:30.529090 24335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1308b5fb7ff942a69ffe1f28a83642bb" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
I20260807 11:08:30.530237 24335 raft_consensus.cc:3037] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:30.534476 24578 consensus_queue.cc:1035] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260807 11:08:30.548447 24512 tablet_bootstrap.cc:492] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d: No bootstrap required, opened a new log
I20260807 11:08:30.549005 24512 ts_tablet_manager.cc:1397] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d: Time spent bootstrapping tablet: real 0.225s user 0.027s sys 0.037s
I20260807 11:08:30.543623 24574 raft_consensus.cc:482] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:30.551120 24574 raft_consensus.cc:504] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:30.553089 24512 raft_consensus.cc:348] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:30.554054 24512 raft_consensus.cc:374] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:30.554440 24512 raft_consensus.cc:729] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212ebf83bd07438a82bd50130d0f427d, State: Initialized, Role: FOLLOWER
W20260807 11:08:30.554642 24142 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 11:08:30.544641 24221 raft_consensus.cc:2454] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212ebf83bd07438a82bd50130d0f427d in term 1.
I20260807 11:08:30.555544 24512 consensus_queue.cc:260] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:30.559930 24512 ts_tablet_manager.cc:1428] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d: Time spent starting tablet: real 0.010s user 0.005s sys 0.003s
I20260807 11:08:30.561234 24442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0a8dfa345949b192adaddd29002763" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212ebf83bd07438a82bd50130d0f427d" is_pre_election: true
I20260807 11:08:30.563297 24393 leader_election.cc:302] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d, 625cae382881486e90d3a22d03b7ef7a; no voters:
I20260807 11:08:30.564795 24578 raft_consensus.cc:2781] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:08:30.565675 24578 raft_consensus.cc:686] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [term 1 LEADER]: Becoming Leader. State: Replica: 212ebf83bd07438a82bd50130d0f427d, State: Running, Role: LEADER
I20260807 11:08:30.567301 24578 consensus_queue.cc:237] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:30.581305 24221 raft_consensus.cc:3037] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:30.556918 24334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0a8dfa345949b192adaddd29002763" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79" is_pre_election: true
I20260807 11:08:30.588250 24175 leader_election.cc:302] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 625cae382881486e90d3a22d03b7ef7a; no voters: 212ebf83bd07438a82bd50130d0f427d, 2e04f9d945c14636bc08ef0f3183fa79
I20260807 11:08:30.568565 24574 leader_election.cc:288] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:30.624522 24509 tablet_bootstrap.cc:492] T bc59e5e4c27b459e938d8942f3ff65c3 P 2e04f9d945c14636bc08ef0f3183fa79: No bootstrap required, opened a new log
I20260807 11:08:30.626824 24509 ts_tablet_manager.cc:1397] T bc59e5e4c27b459e938d8942f3ff65c3 P 2e04f9d945c14636bc08ef0f3183fa79: Time spent bootstrapping tablet: real 0.334s user 0.029s sys 0.057s
I20260807 11:08:30.634279 24509 raft_consensus.cc:348] T bc59e5e4c27b459e938d8942f3ff65c3 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:30.635131 24509 raft_consensus.cc:374] T bc59e5e4c27b459e938d8942f3ff65c3 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:30.635607 24509 raft_consensus.cc:729] T bc59e5e4c27b459e938d8942f3ff65c3 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e04f9d945c14636bc08ef0f3183fa79, State: Initialized, Role: FOLLOWER
I20260807 11:08:30.636497 24509 consensus_queue.cc:260] T bc59e5e4c27b459e938d8942f3ff65c3 P 2e04f9d945c14636bc08ef0f3183fa79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:30.639006 24509 ts_tablet_manager.cc:1428] T bc59e5e4c27b459e938d8942f3ff65c3 P 2e04f9d945c14636bc08ef0f3183fa79: Time spent starting tablet: real 0.012s user 0.004s sys 0.002s
I20260807 11:08:30.641906 24335 raft_consensus.cc:2454] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212ebf83bd07438a82bd50130d0f427d in term 1.
I20260807 11:08:30.655146 24574 consensus_queue.cc:1035] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:08:30.656370 24605 raft_consensus.cc:2726] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:08:30.686653 24077 catalog_manager.cc:5495] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d reported cstate change: term changed from 0 to 1, leader changed from <none> to 212ebf83bd07438a82bd50130d0f427d (127.23.120.131). New cstate: current_term: 1 leader_uuid: "212ebf83bd07438a82bd50130d0f427d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: HEALTHY } } }
I20260807 11:08:30.701552 24574 consensus_queue.cc:1035] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:08:30.712719 24578 raft_consensus.cc:482] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:30.713446 24578 raft_consensus.cc:504] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:30.718308 24221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143ad113a591461abce196dc159fa9f7" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625cae382881486e90d3a22d03b7ef7a" is_pre_election: true
I20260807 11:08:30.723603 24578 leader_election.cc:288] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129)
I20260807 11:08:30.723960 24334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143ad113a591461abce196dc159fa9f7" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79" is_pre_election: true
I20260807 11:08:30.726830 24399 leader_election.cc:302] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d; no voters: 2e04f9d945c14636bc08ef0f3183fa79, 625cae382881486e90d3a22d03b7ef7a
I20260807 11:08:30.729244 24578 raft_consensus.cc:2726] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:08:30.753403 24620 consensus_queue.cc:1035] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:08:30.769553 24588 consensus_queue.cc:1035] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260807 11:08:30.795300 24443 raft_consensus.cc:3037] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:30.835645 24334 raft_consensus.cc:3037] T bc59e5e4c27b459e938d8942f3ff65c3 P 2e04f9d945c14636bc08ef0f3183fa79 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:31.102042 24585 consensus_queue.cc:1035] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [LEADER]: Connected to new peer: Peer: permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:08:31.120708 24618 consensus_queue.cc:1035] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:08:31.179781 24443 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9f8dec87030d45c9a4b51dd3842292d1"
dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
mode: GRACEFUL
new_leader_uuid: "625cae382881486e90d3a22d03b7ef7a"
from {username='slave'} at 127.0.0.1:41064
I20260807 11:08:31.180569 24443 raft_consensus.cc:595] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [term 1 LEADER]: Received request to transfer leadership to TS 625cae382881486e90d3a22d03b7ef7a
I20260807 11:08:31.207234 24334 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b0993a417f1b4e32ab72ff3d30fbacef"
dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
mode: GRACEFUL
new_leader_uuid: "625cae382881486e90d3a22d03b7ef7a"
from {username='slave'} at 127.0.0.1:53632
I20260807 11:08:31.207796 24334 raft_consensus.cc:595] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 LEADER]: Received request to transfer leadership to TS 625cae382881486e90d3a22d03b7ef7a
I20260807 11:08:31.213495 24443 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1308b5fb7ff942a69ffe1f28a83642bb"
dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
mode: GRACEFUL
new_leader_uuid: "625cae382881486e90d3a22d03b7ef7a"
from {username='slave'} at 127.0.0.1:41064
I20260807 11:08:31.214030 24443 raft_consensus.cc:595] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [term 1 LEADER]: Received request to transfer leadership to TS 625cae382881486e90d3a22d03b7ef7a
I20260807 11:08:31.233664 24221 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "143ad113a591461abce196dc159fa9f7"
dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
mode: GRACEFUL
new_leader_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
from {username='slave'} at 127.0.0.1:39170
I20260807 11:08:31.234330 24221 raft_consensus.cc:595] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [term 1 LEADER]: Received request to transfer leadership to TS 2e04f9d945c14636bc08ef0f3183fa79
I20260807 11:08:31.239699 24443 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1d0a8dfa345949b192adaddd29002763"
dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
mode: GRACEFUL
new_leader_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
from {username='slave'} at 127.0.0.1:41064
I20260807 11:08:31.240228 24443 raft_consensus.cc:595] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [term 1 LEADER]: Received request to transfer leadership to TS 2e04f9d945c14636bc08ef0f3183fa79
I20260807 11:08:31.248802 24221 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d9426ebc1a024747ad29dd72e9a763d4"
dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
mode: GRACEFUL
new_leader_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
from {username='slave'} at 127.0.0.1:39170
I20260807 11:08:31.249449 24221 raft_consensus.cc:595] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [term 1 LEADER]: Received request to transfer leadership to TS 2e04f9d945c14636bc08ef0f3183fa79
I20260807 11:08:31.254618 24221 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bc59e5e4c27b459e938d8942f3ff65c3"
dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
mode: GRACEFUL
new_leader_uuid: "212ebf83bd07438a82bd50130d0f427d"
from {username='slave'} at 127.0.0.1:39170
I20260807 11:08:31.255203 24221 raft_consensus.cc:595] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [term 1 LEADER]: Received request to transfer leadership to TS 212ebf83bd07438a82bd50130d0f427d
I20260807 11:08:31.388195 24644 raft_consensus.cc:979] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a: : Instructing follower 2e04f9d945c14636bc08ef0f3183fa79 to start an election
I20260807 11:08:31.388883 24644 raft_consensus.cc:1067] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [term 1 LEADER]: Signalling peer 2e04f9d945c14636bc08ef0f3183fa79 to start an election
I20260807 11:08:31.390899 24334 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "143ad113a591461abce196dc159fa9f7"
dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
from {username='slave'} at 127.0.0.1:53624
I20260807 11:08:31.391541 24334 raft_consensus.cc:482] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:08:31.391947 24334 raft_consensus.cc:3037] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:31.399305 24334 raft_consensus.cc:504] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:31.401453 24334 leader_election.cc:288] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 2 election: Requested vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:31.402462 24221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143ad113a591461abce196dc159fa9f7" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
I20260807 11:08:31.402467 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143ad113a591461abce196dc159fa9f7" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
I20260807 11:08:31.403164 24221 raft_consensus.cc:3032] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:08:31.403539 24221 raft_consensus.cc:729] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 625cae382881486e90d3a22d03b7ef7a, State: Running, Role: LEADER
I20260807 11:08:31.403586 24443 raft_consensus.cc:3037] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:31.404397 24221 consensus_queue.cc:260] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:31.405807 24221 raft_consensus.cc:3037] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:31.412731 24443 raft_consensus.cc:2454] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e04f9d945c14636bc08ef0f3183fa79 in term 2.
I20260807 11:08:31.414222 24285 leader_election.cc:302] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d, 2e04f9d945c14636bc08ef0f3183fa79; no voters:
I20260807 11:08:31.415120 24221 raft_consensus.cc:2454] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e04f9d945c14636bc08ef0f3183fa79 in term 2.
I20260807 11:08:31.416915 24645 raft_consensus.cc:979] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79: : Instructing follower 625cae382881486e90d3a22d03b7ef7a to start an election
I20260807 11:08:31.417049 24588 raft_consensus.cc:2781] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:08:31.417505 24645 raft_consensus.cc:1067] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 LEADER]: Signalling peer 625cae382881486e90d3a22d03b7ef7a to start an election
I20260807 11:08:31.417598 24588 raft_consensus.cc:686] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 LEADER]: Becoming Leader. State: Replica: 2e04f9d945c14636bc08ef0f3183fa79, State: Running, Role: LEADER
I20260807 11:08:31.419426 24221 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b0993a417f1b4e32ab72ff3d30fbacef"
dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
from {username='slave'} at 127.0.0.1:39162
I20260807 11:08:31.419426 24588 consensus_queue.cc:237] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:31.420176 24221 raft_consensus.cc:482] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:08:31.420627 24221 raft_consensus.cc:3037] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:31.426589 24221 raft_consensus.cc:504] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:31.428498 24077 catalog_manager.cc:5495] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 reported cstate change: term changed from 1 to 2, leader changed from 625cae382881486e90d3a22d03b7ef7a (127.23.120.129) to 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130). New cstate: current_term: 2 leader_uuid: "2e04f9d945c14636bc08ef0f3183fa79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: UNKNOWN } } }
I20260807 11:08:31.429266 24334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0993a417f1b4e32ab72ff3d30fbacef" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
I20260807 11:08:31.429787 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0993a417f1b4e32ab72ff3d30fbacef" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
I20260807 11:08:31.429925 24334 raft_consensus.cc:3032] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:08:31.430423 24443 raft_consensus.cc:3037] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:31.430435 24334 raft_consensus.cc:729] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2e04f9d945c14636bc08ef0f3183fa79, State: Running, Role: LEADER
I20260807 11:08:31.429181 24221 leader_election.cc:288] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 2 election: Requested vote from peers 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:31.431385 24334 consensus_queue.cc:260] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:31.433458 24334 raft_consensus.cc:3037] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:31.436550 24443 raft_consensus.cc:2454] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625cae382881486e90d3a22d03b7ef7a in term 2.
I20260807 11:08:31.437646 24174 leader_election.cc:302] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d, 625cae382881486e90d3a22d03b7ef7a; no voters:
I20260807 11:08:31.438537 24643 raft_consensus.cc:2781] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:08:31.438872 24334 raft_consensus.cc:2454] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625cae382881486e90d3a22d03b7ef7a in term 2.
I20260807 11:08:31.439054 24643 raft_consensus.cc:686] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 2 LEADER]: Becoming Leader. State: Replica: 625cae382881486e90d3a22d03b7ef7a, State: Running, Role: LEADER
I20260807 11:08:31.439982 24643 consensus_queue.cc:237] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:31.445920 24077 catalog_manager.cc:5495] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a reported cstate change: term changed from 1 to 2, leader changed from 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130) to 625cae382881486e90d3a22d03b7ef7a (127.23.120.129). New cstate: current_term: 2 leader_uuid: "625cae382881486e90d3a22d03b7ef7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: UNKNOWN } } }
I20260807 11:08:31.489357 24643 raft_consensus.cc:979] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a: : Instructing follower 2e04f9d945c14636bc08ef0f3183fa79 to start an election
I20260807 11:08:31.489820 24644 raft_consensus.cc:1067] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [term 1 LEADER]: Signalling peer 2e04f9d945c14636bc08ef0f3183fa79 to start an election
I20260807 11:08:31.491110 24334 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d9426ebc1a024747ad29dd72e9a763d4"
dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
from {username='slave'} at 127.0.0.1:53624
I20260807 11:08:31.491532 24334 raft_consensus.cc:482] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:08:31.491782 24334 raft_consensus.cc:3037] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:31.496234 24334 raft_consensus.cc:504] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:31.497699 24334 leader_election.cc:288] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 2 election: Requested vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:31.498476 24221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9426ebc1a024747ad29dd72e9a763d4" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
I20260807 11:08:31.498713 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9426ebc1a024747ad29dd72e9a763d4" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
I20260807 11:08:31.499099 24221 raft_consensus.cc:3032] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:08:31.499220 24443 raft_consensus.cc:3037] T d9426ebc1a024747ad29dd72e9a763d4 P 212ebf83bd07438a82bd50130d0f427d [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:31.499454 24221 raft_consensus.cc:729] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 625cae382881486e90d3a22d03b7ef7a, State: Running, Role: LEADER
I20260807 11:08:31.500219 24221 consensus_queue.cc:260] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:31.501456 24221 raft_consensus.cc:3037] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:31.505623 24443 raft_consensus.cc:2454] T d9426ebc1a024747ad29dd72e9a763d4 P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e04f9d945c14636bc08ef0f3183fa79 in term 2.
I20260807 11:08:31.506584 24285 leader_election.cc:302] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d, 2e04f9d945c14636bc08ef0f3183fa79; no voters:
I20260807 11:08:31.507237 24588 raft_consensus.cc:2781] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:08:31.507315 24221 raft_consensus.cc:2454] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e04f9d945c14636bc08ef0f3183fa79 in term 2.
I20260807 11:08:31.507664 24588 raft_consensus.cc:686] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 LEADER]: Becoming Leader. State: Replica: 2e04f9d945c14636bc08ef0f3183fa79, State: Running, Role: LEADER
I20260807 11:08:31.508446 24588 consensus_queue.cc:237] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:31.515399 24077 catalog_manager.cc:5495] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79 reported cstate change: term changed from 1 to 2, leader changed from 625cae382881486e90d3a22d03b7ef7a (127.23.120.129) to 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130). New cstate: current_term: 2 leader_uuid: "2e04f9d945c14636bc08ef0f3183fa79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: UNKNOWN } } }
I20260807 11:08:31.580528 24578 raft_consensus.cc:979] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d: : Instructing follower 625cae382881486e90d3a22d03b7ef7a to start an election
I20260807 11:08:31.581035 24618 raft_consensus.cc:1067] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [term 1 LEADER]: Signalling peer 625cae382881486e90d3a22d03b7ef7a to start an election
I20260807 11:08:31.582501 24221 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9f8dec87030d45c9a4b51dd3842292d1"
dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
from {username='slave'} at 127.0.0.1:39160
I20260807 11:08:31.582979 24221 raft_consensus.cc:482] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:08:31.583292 24221 raft_consensus.cc:3037] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:31.587590 24221 raft_consensus.cc:504] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:31.589206 24221 leader_election.cc:288] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 2 election: Requested vote from peers 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:31.590060 24334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8dec87030d45c9a4b51dd3842292d1" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
I20260807 11:08:31.590217 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8dec87030d45c9a4b51dd3842292d1" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
I20260807 11:08:31.590663 24334 raft_consensus.cc:3037] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:31.590687 24443 raft_consensus.cc:3032] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:08:31.591225 24443 raft_consensus.cc:729] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 212ebf83bd07438a82bd50130d0f427d, State: Running, Role: LEADER
I20260807 11:08:31.592002 24443 consensus_queue.cc:260] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:31.593247 24443 raft_consensus.cc:3037] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:31.597209 24334 raft_consensus.cc:2454] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625cae382881486e90d3a22d03b7ef7a in term 2.
I20260807 11:08:31.598335 24175 leader_election.cc:302] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e04f9d945c14636bc08ef0f3183fa79, 625cae382881486e90d3a22d03b7ef7a; no voters:
I20260807 11:08:31.598990 24644 raft_consensus.cc:2781] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:08:31.599515 24443 raft_consensus.cc:2454] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625cae382881486e90d3a22d03b7ef7a in term 2.
I20260807 11:08:31.599412 24644 raft_consensus.cc:686] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [term 2 LEADER]: Becoming Leader. State: Replica: 625cae382881486e90d3a22d03b7ef7a, State: Running, Role: LEADER
I20260807 11:08:31.600849 24574 raft_consensus.cc:979] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a: : Instructing follower 212ebf83bd07438a82bd50130d0f427d to start an election
I20260807 11:08:31.601430 24643 raft_consensus.cc:1067] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [term 1 LEADER]: Signalling peer 212ebf83bd07438a82bd50130d0f427d to start an election
I20260807 11:08:31.600852 24644 consensus_queue.cc:237] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:31.602886 24443 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bc59e5e4c27b459e938d8942f3ff65c3"
dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
from {username='slave'} at 127.0.0.1:41058
I20260807 11:08:31.603498 24443 raft_consensus.cc:482] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:08:31.603842 24443 raft_consensus.cc:3037] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:31.608927 24077 catalog_manager.cc:5495] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a reported cstate change: term changed from 1 to 2, leader changed from 212ebf83bd07438a82bd50130d0f427d (127.23.120.131) to 625cae382881486e90d3a22d03b7ef7a (127.23.120.129). New cstate: current_term: 2 leader_uuid: "625cae382881486e90d3a22d03b7ef7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: UNKNOWN } } }
I20260807 11:08:31.609436 24443 raft_consensus.cc:504] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:31.611377 24443 leader_election.cc:288] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 2 election: Requested vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129)
I20260807 11:08:31.612056 24221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc59e5e4c27b459e938d8942f3ff65c3" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
I20260807 11:08:31.617564 24221 raft_consensus.cc:3032] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:08:31.612423 24334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc59e5e4c27b459e938d8942f3ff65c3" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
I20260807 11:08:31.617996 24221 raft_consensus.cc:729] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 625cae382881486e90d3a22d03b7ef7a, State: Running, Role: LEADER
I20260807 11:08:31.618243 24334 raft_consensus.cc:3037] T bc59e5e4c27b459e938d8942f3ff65c3 P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:31.618871 24221 consensus_queue.cc:260] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:31.620107 24221 raft_consensus.cc:3037] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:31.624502 24578 raft_consensus.cc:979] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d: : Instructing follower 2e04f9d945c14636bc08ef0f3183fa79 to start an election
I20260807 11:08:31.624933 24578 raft_consensus.cc:1067] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [term 1 LEADER]: Signalling peer 2e04f9d945c14636bc08ef0f3183fa79 to start an election
I20260807 11:08:31.624974 24334 raft_consensus.cc:2454] T bc59e5e4c27b459e938d8942f3ff65c3 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212ebf83bd07438a82bd50130d0f427d in term 2.
I20260807 11:08:31.626096 24399 leader_election.cc:302] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d, 2e04f9d945c14636bc08ef0f3183fa79; no voters:
I20260807 11:08:31.626509 24334 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1d0a8dfa345949b192adaddd29002763"
dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
from {username='slave'} at 127.0.0.1:53610
I20260807 11:08:31.626600 24221 raft_consensus.cc:2454] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212ebf83bd07438a82bd50130d0f427d in term 2.
I20260807 11:08:31.626792 24578 raft_consensus.cc:2781] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:08:31.627122 24334 raft_consensus.cc:482] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:08:31.627238 24578 raft_consensus.cc:686] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d [term 2 LEADER]: Becoming Leader. State: Replica: 212ebf83bd07438a82bd50130d0f427d, State: Running, Role: LEADER
I20260807 11:08:31.627528 24334 raft_consensus.cc:3037] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:31.628098 24578 consensus_queue.cc:237] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:31.634019 24334 raft_consensus.cc:504] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:31.635845 24334 leader_election.cc:288] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 2 election: Requested vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:31.636515 24221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0a8dfa345949b192adaddd29002763" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
I20260807 11:08:31.636950 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0a8dfa345949b192adaddd29002763" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
I20260807 11:08:31.637084 24221 raft_consensus.cc:3037] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:31.637373 24443 raft_consensus.cc:3032] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:08:31.637583 24443 raft_consensus.cc:729] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 212ebf83bd07438a82bd50130d0f427d, State: Running, Role: LEADER
I20260807 11:08:31.636137 24077 catalog_manager.cc:5495] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d reported cstate change: term changed from 1 to 2, leader changed from 625cae382881486e90d3a22d03b7ef7a (127.23.120.129) to 212ebf83bd07438a82bd50130d0f427d (127.23.120.131). New cstate: current_term: 2 leader_uuid: "212ebf83bd07438a82bd50130d0f427d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: HEALTHY } } }
I20260807 11:08:31.638245 24443 consensus_queue.cc:260] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:31.639406 24443 raft_consensus.cc:3037] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:31.642373 24221 raft_consensus.cc:2454] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e04f9d945c14636bc08ef0f3183fa79 in term 2.
I20260807 11:08:31.643472 24283 leader_election.cc:302] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e04f9d945c14636bc08ef0f3183fa79, 625cae382881486e90d3a22d03b7ef7a; no voters:
I20260807 11:08:31.644877 24588 raft_consensus.cc:2781] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:08:31.645287 24588 raft_consensus.cc:686] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 LEADER]: Becoming Leader. State: Replica: 2e04f9d945c14636bc08ef0f3183fa79, State: Running, Role: LEADER
I20260807 11:08:31.645457 24443 raft_consensus.cc:2454] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e04f9d945c14636bc08ef0f3183fa79 in term 2.
I20260807 11:08:31.646061 24588 consensus_queue.cc:237] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:31.652524 24077 catalog_manager.cc:5495] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 reported cstate change: term changed from 1 to 2, leader changed from 212ebf83bd07438a82bd50130d0f427d (127.23.120.131) to 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130). New cstate: current_term: 2 leader_uuid: "2e04f9d945c14636bc08ef0f3183fa79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: UNKNOWN } } }
I20260807 11:08:31.653254 24618 raft_consensus.cc:979] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d: : Instructing follower 625cae382881486e90d3a22d03b7ef7a to start an election
I20260807 11:08:31.653905 24578 raft_consensus.cc:1067] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [term 1 LEADER]: Signalling peer 625cae382881486e90d3a22d03b7ef7a to start an election
I20260807 11:08:31.655224 24221 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1308b5fb7ff942a69ffe1f28a83642bb"
dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
from {username='slave'} at 127.0.0.1:39160
I20260807 11:08:31.655758 24221 raft_consensus.cc:482] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:08:31.656095 24221 raft_consensus.cc:3037] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:31.661525 24221 raft_consensus.cc:504] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:31.662858 24221 leader_election.cc:288] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 2 election: Requested vote from peers 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:31.663671 24334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1308b5fb7ff942a69ffe1f28a83642bb" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
I20260807 11:08:31.663769 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1308b5fb7ff942a69ffe1f28a83642bb" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
I20260807 11:08:31.664199 24334 raft_consensus.cc:3037] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:31.664358 24443 raft_consensus.cc:3032] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:08:31.664755 24443 raft_consensus.cc:729] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 212ebf83bd07438a82bd50130d0f427d, State: Running, Role: LEADER
I20260807 11:08:31.665380 24443 consensus_queue.cc:260] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:31.666226 24443 raft_consensus.cc:3037] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:31.668731 24334 raft_consensus.cc:2454] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625cae382881486e90d3a22d03b7ef7a in term 2.
I20260807 11:08:31.669620 24175 leader_election.cc:302] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e04f9d945c14636bc08ef0f3183fa79, 625cae382881486e90d3a22d03b7ef7a; no voters:
I20260807 11:08:31.670286 24644 raft_consensus.cc:2781] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:08:31.670706 24644 raft_consensus.cc:686] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [term 2 LEADER]: Becoming Leader. State: Replica: 625cae382881486e90d3a22d03b7ef7a, State: Running, Role: LEADER
I20260807 11:08:31.671146 24443 raft_consensus.cc:2454] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625cae382881486e90d3a22d03b7ef7a in term 2.
I20260807 11:08:31.671381 24644 consensus_queue.cc:237] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:31.677031 24077 catalog_manager.cc:5495] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a reported cstate change: term changed from 1 to 2, leader changed from 212ebf83bd07438a82bd50130d0f427d (127.23.120.131) to 625cae382881486e90d3a22d03b7ef7a (127.23.120.129). New cstate: current_term: 2 leader_uuid: "625cae382881486e90d3a22d03b7ef7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: UNKNOWN } } }
I20260807 11:08:31.840904 24443 raft_consensus.cc:1261] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Refusing update from remote peer 2e04f9d945c14636bc08ef0f3183fa79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:08:31.842555 24647 consensus_queue.cc:1035] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:08:31.854598 24221 raft_consensus.cc:1261] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Refusing update from remote peer 2e04f9d945c14636bc08ef0f3183fa79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:08:31.856117 24647 consensus_queue.cc:1035] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:08:31.982113 24443 raft_consensus.cc:1261] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Refusing update from remote peer 625cae382881486e90d3a22d03b7ef7a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:08:31.983459 24644 consensus_queue.cc:1035] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:08:31.991953 24334 raft_consensus.cc:1261] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Refusing update from remote peer 625cae382881486e90d3a22d03b7ef7a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:08:31.994639 24644 consensus_queue.cc:1035] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:08:31.995980 24443 raft_consensus.cc:1261] T d9426ebc1a024747ad29dd72e9a763d4 P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Refusing update from remote peer 2e04f9d945c14636bc08ef0f3183fa79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:08:31.997720 24647 consensus_queue.cc:1035] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:08:32.011411 24221 raft_consensus.cc:1261] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Refusing update from remote peer 2e04f9d945c14636bc08ef0f3183fa79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:08:32.013213 24645 consensus_queue.cc:1035] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:08:32.030808 24221 raft_consensus.cc:1261] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Refusing update from remote peer 212ebf83bd07438a82bd50130d0f427d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:08:32.032214 24578 consensus_queue.cc:1035] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d [LEADER]: Connected to new peer: Peer: permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:08:32.041013 24337 raft_consensus.cc:1261] T bc59e5e4c27b459e938d8942f3ff65c3 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Refusing update from remote peer 212ebf83bd07438a82bd50130d0f427d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:08:32.042567 24677 consensus_queue.cc:1035] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:08:32.087168 24443 raft_consensus.cc:1261] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Refusing update from remote peer 2e04f9d945c14636bc08ef0f3183fa79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:08:32.088526 24645 consensus_queue.cc:1035] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:08:32.099534 24221 raft_consensus.cc:1261] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Refusing update from remote peer 2e04f9d945c14636bc08ef0f3183fa79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:08:32.101488 24588 consensus_queue.cc:1035] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:08:32.116418 24443 raft_consensus.cc:1261] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Refusing update from remote peer 625cae382881486e90d3a22d03b7ef7a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:08:32.117573 24644 consensus_queue.cc:1035] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:08:32.125298 24337 raft_consensus.cc:1261] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Refusing update from remote peer 625cae382881486e90d3a22d03b7ef7a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:08:32.127031 24574 consensus_queue.cc:1035] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:08:32.192479 24337 raft_consensus.cc:1261] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Refusing update from remote peer 625cae382881486e90d3a22d03b7ef7a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:08:32.193784 24574 consensus_queue.cc:1035] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:08:32.201570 24443 raft_consensus.cc:1261] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Refusing update from remote peer 625cae382881486e90d3a22d03b7ef7a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:08:32.203213 24644 consensus_queue.cc:1035] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:08:34.297001 24034 auto_leader_rebalancer-test.cc:269] The leader distribution is
I20260807 11:08:34.298302 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:08:34.325665 24334 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "143ad113a591461abce196dc159fa9f7"
dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
mode: GRACEFUL
new_leader_uuid: "212ebf83bd07438a82bd50130d0f427d"
from {username='slave'} at 127.0.0.1:53646
I20260807 11:08:34.326776 24334 raft_consensus.cc:595] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 LEADER]: Received request to transfer leadership to TS 212ebf83bd07438a82bd50130d0f427d
I20260807 11:08:34.342265 24221 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b0993a417f1b4e32ab72ff3d30fbacef"
dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
mode: GRACEFUL
new_leader_uuid: "212ebf83bd07438a82bd50130d0f427d"
from {username='slave'} at 127.0.0.1:39178
I20260807 11:08:34.342796 24221 raft_consensus.cc:595] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 2 LEADER]: Received request to transfer leadership to TS 212ebf83bd07438a82bd50130d0f427d
I20260807 11:08:34.344061 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260807 11:08:34.344643 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:08:34.537595 24700 raft_consensus.cc:979] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a: : Instructing follower 212ebf83bd07438a82bd50130d0f427d to start an election
I20260807 11:08:34.538347 24694 raft_consensus.cc:1067] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 2 LEADER]: Signalling peer 212ebf83bd07438a82bd50130d0f427d to start an election
I20260807 11:08:34.539951 24443 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b0993a417f1b4e32ab72ff3d30fbacef"
dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
from {username='slave'} at 127.0.0.1:41058
I20260807 11:08:34.540406 24443 raft_consensus.cc:482] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:08:34.540711 24443 raft_consensus.cc:3037] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:08:34.545745 24443 raft_consensus.cc:504] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:34.547315 24443 leader_election.cc:288] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 3 election: Requested vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129)
I20260807 11:08:34.548363 24221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0993a417f1b4e32ab72ff3d30fbacef" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
I20260807 11:08:34.548452 24334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0993a417f1b4e32ab72ff3d30fbacef" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
I20260807 11:08:34.549263 24221 raft_consensus.cc:3032] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:08:34.549292 24334 raft_consensus.cc:3037] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:08:34.549685 24221 raft_consensus.cc:729] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 625cae382881486e90d3a22d03b7ef7a, State: Running, Role: LEADER
I20260807 11:08:34.550457 24221 consensus_queue.cc:260] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:34.551441 24221 raft_consensus.cc:3037] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:08:34.554258 24334 raft_consensus.cc:2454] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212ebf83bd07438a82bd50130d0f427d in term 3.
I20260807 11:08:34.555295 24399 leader_election.cc:302] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d, 2e04f9d945c14636bc08ef0f3183fa79; no voters:
I20260807 11:08:34.556090 24691 raft_consensus.cc:2781] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:08:34.556443 24221 raft_consensus.cc:2454] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212ebf83bd07438a82bd50130d0f427d in term 3.
I20260807 11:08:34.557407 24691 raft_consensus.cc:686] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 3 LEADER]: Becoming Leader. State: Replica: 212ebf83bd07438a82bd50130d0f427d, State: Running, Role: LEADER
I20260807 11:08:34.558212 24691 consensus_queue.cc:237] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:34.565145 24078 catalog_manager.cc:5495] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d reported cstate change: term changed from 2 to 3, leader changed from 625cae382881486e90d3a22d03b7ef7a (127.23.120.129) to 212ebf83bd07438a82bd50130d0f427d (127.23.120.131). New cstate: current_term: 3 leader_uuid: "212ebf83bd07438a82bd50130d0f427d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: HEALTHY } } }
I20260807 11:08:34.762564 24647 raft_consensus.cc:979] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79: : Instructing follower 212ebf83bd07438a82bd50130d0f427d to start an election
I20260807 11:08:34.763113 24711 raft_consensus.cc:1067] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 LEADER]: Signalling peer 212ebf83bd07438a82bd50130d0f427d to start an election
I20260807 11:08:34.764621 24443 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "143ad113a591461abce196dc159fa9f7"
dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
from {username='slave'} at 127.0.0.1:41062
I20260807 11:08:34.765188 24443 raft_consensus.cc:482] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:08:34.765486 24443 raft_consensus.cc:3037] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:08:34.769686 24443 raft_consensus.cc:504] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:34.771015 24443 leader_election.cc:288] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 3 election: Requested vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129)
I20260807 11:08:34.772132 24221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143ad113a591461abce196dc159fa9f7" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
I20260807 11:08:34.772210 24334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143ad113a591461abce196dc159fa9f7" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
I20260807 11:08:34.772732 24221 raft_consensus.cc:3037] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:08:34.772841 24334 raft_consensus.cc:3032] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:08:34.773202 24334 raft_consensus.cc:729] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2e04f9d945c14636bc08ef0f3183fa79, State: Running, Role: LEADER
I20260807 11:08:34.773896 24334 consensus_queue.cc:260] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:34.774833 24334 raft_consensus.cc:3037] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:08:34.777091 24221 raft_consensus.cc:2454] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212ebf83bd07438a82bd50130d0f427d in term 3.
I20260807 11:08:34.778151 24393 leader_election.cc:302] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d, 625cae382881486e90d3a22d03b7ef7a; no voters:
I20260807 11:08:34.778945 24691 raft_consensus.cc:2781] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:08:34.779363 24691 raft_consensus.cc:686] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [term 3 LEADER]: Becoming Leader. State: Replica: 212ebf83bd07438a82bd50130d0f427d, State: Running, Role: LEADER
I20260807 11:08:34.779439 24334 raft_consensus.cc:2454] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212ebf83bd07438a82bd50130d0f427d in term 3.
I20260807 11:08:34.780297 24691 consensus_queue.cc:237] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:34.787415 24078 catalog_manager.cc:5495] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d reported cstate change: term changed from 2 to 3, leader changed from 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130) to 212ebf83bd07438a82bd50130d0f427d (127.23.120.131). New cstate: current_term: 3 leader_uuid: "212ebf83bd07438a82bd50130d0f427d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: HEALTHY } } }
I20260807 11:08:35.033736 24221 raft_consensus.cc:1261] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 3 FOLLOWER]: Refusing update from remote peer 212ebf83bd07438a82bd50130d0f427d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:08:35.034981 24691 consensus_queue.cc:1035] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [LEADER]: Connected to new peer: Peer: permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:08:35.045876 24334 raft_consensus.cc:1261] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 FOLLOWER]: Refusing update from remote peer 212ebf83bd07438a82bd50130d0f427d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:08:35.047437 24701 consensus_queue.cc:1035] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:08:35.183732 24334 raft_consensus.cc:1261] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 FOLLOWER]: Refusing update from remote peer 212ebf83bd07438a82bd50130d0f427d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:08:35.185122 24691 consensus_queue.cc:1035] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:08:35.194201 24221 raft_consensus.cc:1261] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [term 3 FOLLOWER]: Refusing update from remote peer 212ebf83bd07438a82bd50130d0f427d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:08:35.195689 24701 consensus_queue.cc:1035] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [LEADER]: Connected to new peer: Peer: permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:08:35.345568 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:08:35.350123 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:08:35.350519 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:08:36.351282 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:08:36.356753 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:08:36.357175 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:08:37.358006 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:08:37.363480 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:08:37.364007 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:08:38.364835 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:08:38.369454 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:08:38.369840 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:08:39.370672 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:08:39.375506 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:08:39.375921 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:08:40.376652 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:08:40.382436 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:08:40.382809 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:08:41.461308 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:08:41.465893 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:08:41.466202 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260807 11:08:41.829177 24117 debug-util.cc:398] Leaking SignalData structure 0x7b08000607a0 after lost signal to thread 24038
W20260807 11:08:41.829957 24117 debug-util.cc:398] Leaking SignalData structure 0x7b08000600a0 after lost signal to thread 24122
W20260807 11:08:41.830744 24117 debug-util.cc:398] Leaking SignalData structure 0x7b0800146920 after lost signal to thread 24261
W20260807 11:08:41.831440 24117 debug-util.cc:398] Leaking SignalData structure 0x7b08001b2a80 after lost signal to thread 24369
W20260807 11:08:41.832120 24117 debug-util.cc:398] Leaking SignalData structure 0x7b08001f8240 after lost signal to thread 24477
I20260807 11:08:42.466982 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:08:42.472446 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:08:42.472898 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:08:43.473997 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:08:43.478967 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:08:43.479365 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:08:44.514448 24034 auto_leader_rebalancer-test.cc:288] The leader distribution is
I20260807 11:08:44.521286 24228 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9f8dec87030d45c9a4b51dd3842292d1"
dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
mode: GRACEFUL
new_leader_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
from {username='slave'} at 127.0.0.1:39170
I20260807 11:08:44.523399 24228 raft_consensus.cc:595] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [term 2 LEADER]: Received request to transfer leadership to TS 2e04f9d945c14636bc08ef0f3183fa79
I20260807 11:08:44.530885 24228 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5a1d2a0a8e38476ea55fdd85717cf7ea"
dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
mode: GRACEFUL
new_leader_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
from {username='slave'} at 127.0.0.1:39170
I20260807 11:08:44.531553 24228 raft_consensus.cc:595] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [term 1 LEADER]: Received request to transfer leadership to TS 2e04f9d945c14636bc08ef0f3183fa79
I20260807 11:08:44.537309 24442 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b0993a417f1b4e32ab72ff3d30fbacef"
dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
mode: GRACEFUL
new_leader_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
from {username='slave'} at 127.0.0.1:41064
I20260807 11:08:44.538062 24442 raft_consensus.cc:595] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 3 LEADER]: Received request to transfer leadership to TS 2e04f9d945c14636bc08ef0f3183fa79
I20260807 11:08:44.544710 24228 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1308b5fb7ff942a69ffe1f28a83642bb"
dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
mode: GRACEFUL
new_leader_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
from {username='slave'} at 127.0.0.1:39170
I20260807 11:08:44.545650 24228 raft_consensus.cc:595] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [term 2 LEADER]: Received request to transfer leadership to TS 2e04f9d945c14636bc08ef0f3183fa79
I20260807 11:08:44.551837 24442 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "143ad113a591461abce196dc159fa9f7"
dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
mode: GRACEFUL
new_leader_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
from {username='slave'} at 127.0.0.1:41064
I20260807 11:08:44.552505 24442 raft_consensus.cc:595] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [term 3 LEADER]: Received request to transfer leadership to TS 2e04f9d945c14636bc08ef0f3183fa79
I20260807 11:08:44.750811 24738 raft_consensus.cc:979] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a: : Instructing follower 2e04f9d945c14636bc08ef0f3183fa79 to start an election
I20260807 11:08:44.752041 24743 raft_consensus.cc:1067] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [term 1 LEADER]: Signalling peer 2e04f9d945c14636bc08ef0f3183fa79 to start an election
I20260807 11:08:44.754873 24333 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5a1d2a0a8e38476ea55fdd85717cf7ea"
dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
from {username='slave'} at 127.0.0.1:53624
I20260807 11:08:44.755595 24333 raft_consensus.cc:482] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:08:44.756198 24333 raft_consensus.cc:3037] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:44.766741 24333 raft_consensus.cc:504] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:44.771034 24333 leader_election.cc:288] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 2 election: Requested vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:44.772253 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1d2a0a8e38476ea55fdd85717cf7ea" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
I20260807 11:08:44.773015 24442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1d2a0a8e38476ea55fdd85717cf7ea" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
I20260807 11:08:44.773769 24228 raft_consensus.cc:3032] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:08:44.774062 24442 raft_consensus.cc:3037] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 212ebf83bd07438a82bd50130d0f427d [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:44.774331 24228 raft_consensus.cc:729] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 625cae382881486e90d3a22d03b7ef7a, State: Running, Role: LEADER
I20260807 11:08:44.775758 24228 consensus_queue.cc:260] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:44.778412 24228 raft_consensus.cc:3037] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:44.789367 24442 raft_consensus.cc:2454] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e04f9d945c14636bc08ef0f3183fa79 in term 2.
I20260807 11:08:44.792229 24285 leader_election.cc:302] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d, 2e04f9d945c14636bc08ef0f3183fa79; no voters:
I20260807 11:08:44.794267 24228 raft_consensus.cc:2454] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e04f9d945c14636bc08ef0f3183fa79 in term 2.
I20260807 11:08:44.794291 24742 raft_consensus.cc:2781] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:08:44.796204 24742 raft_consensus.cc:686] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 LEADER]: Becoming Leader. State: Replica: 2e04f9d945c14636bc08ef0f3183fa79, State: Running, Role: LEADER
I20260807 11:08:44.797576 24742 consensus_queue.cc:237] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:44.821585 24730 raft_consensus.cc:979] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d: : Instructing follower 2e04f9d945c14636bc08ef0f3183fa79 to start an election
I20260807 11:08:44.823143 24750 raft_consensus.cc:1067] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 3 LEADER]: Signalling peer 2e04f9d945c14636bc08ef0f3183fa79 to start an election
I20260807 11:08:44.822965 24078 catalog_manager.cc:5495] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 reported cstate change: term changed from 1 to 2, leader changed from 625cae382881486e90d3a22d03b7ef7a (127.23.120.129) to 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130). New cstate: current_term: 2 leader_uuid: "2e04f9d945c14636bc08ef0f3183fa79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: UNKNOWN } } }
I20260807 11:08:44.826221 24333 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b0993a417f1b4e32ab72ff3d30fbacef"
dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
from {username='slave'} at 127.0.0.1:53610
I20260807 11:08:44.826879 24333 raft_consensus.cc:482] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:08:44.827483 24333 raft_consensus.cc:3037] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:08:44.835839 24333 raft_consensus.cc:504] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:44.838547 24333 leader_election.cc:288] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 4 election: Requested vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:44.840499 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0993a417f1b4e32ab72ff3d30fbacef" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
I20260807 11:08:44.841310 24228 raft_consensus.cc:3037] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:08:44.847663 24442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0993a417f1b4e32ab72ff3d30fbacef" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
I20260807 11:08:44.848913 24442 raft_consensus.cc:3032] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:08:44.849766 24442 raft_consensus.cc:729] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 212ebf83bd07438a82bd50130d0f427d, State: Running, Role: LEADER
I20260807 11:08:44.851222 24442 consensus_queue.cc:260] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:44.853865 24442 raft_consensus.cc:3037] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:08:44.854626 24228 raft_consensus.cc:2454] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e04f9d945c14636bc08ef0f3183fa79 in term 4.
I20260807 11:08:44.857175 24283 leader_election.cc:302] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e04f9d945c14636bc08ef0f3183fa79, 625cae382881486e90d3a22d03b7ef7a; no voters:
I20260807 11:08:44.860659 24744 raft_consensus.cc:2781] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:08:44.862425 24744 raft_consensus.cc:686] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 4 LEADER]: Becoming Leader. State: Replica: 2e04f9d945c14636bc08ef0f3183fa79, State: Running, Role: LEADER
I20260807 11:08:44.864990 24744 consensus_queue.cc:237] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:44.869275 24442 raft_consensus.cc:2454] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e04f9d945c14636bc08ef0f3183fa79 in term 4.
I20260807 11:08:44.869915 24730 raft_consensus.cc:979] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d: : Instructing follower 2e04f9d945c14636bc08ef0f3183fa79 to start an election
I20260807 11:08:44.870645 24750 raft_consensus.cc:1067] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [term 3 LEADER]: Signalling peer 2e04f9d945c14636bc08ef0f3183fa79 to start an election
I20260807 11:08:44.872860 24333 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "143ad113a591461abce196dc159fa9f7"
dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
from {username='slave'} at 127.0.0.1:53610
I20260807 11:08:44.873621 24333 raft_consensus.cc:482] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:08:44.874215 24333 raft_consensus.cc:3037] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:08:44.880584 24078 catalog_manager.cc:5495] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 reported cstate change: term changed from 3 to 4, leader changed from 212ebf83bd07438a82bd50130d0f427d (127.23.120.131) to 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130). New cstate: current_term: 4 leader_uuid: "2e04f9d945c14636bc08ef0f3183fa79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: UNKNOWN } } }
I20260807 11:08:44.884654 24333 raft_consensus.cc:504] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:44.889649 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143ad113a591461abce196dc159fa9f7" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
I20260807 11:08:44.889890 24333 leader_election.cc:288] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 4 election: Requested vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:44.890594 24228 raft_consensus.cc:3037] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:08:44.889740 24442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143ad113a591461abce196dc159fa9f7" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
I20260807 11:08:44.891527 24442 raft_consensus.cc:3032] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:08:44.892060 24442 raft_consensus.cc:729] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 212ebf83bd07438a82bd50130d0f427d, State: Running, Role: LEADER
I20260807 11:08:44.893321 24442 consensus_queue.cc:260] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:44.895380 24442 raft_consensus.cc:3037] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:08:44.900432 24228 raft_consensus.cc:2454] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e04f9d945c14636bc08ef0f3183fa79 in term 4.
I20260807 11:08:44.901830 24283 leader_election.cc:302] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e04f9d945c14636bc08ef0f3183fa79, 625cae382881486e90d3a22d03b7ef7a; no voters:
I20260807 11:08:44.902707 24744 raft_consensus.cc:2781] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:08:44.903352 24744 raft_consensus.cc:686] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [term 4 LEADER]: Becoming Leader. State: Replica: 2e04f9d945c14636bc08ef0f3183fa79, State: Running, Role: LEADER
I20260807 11:08:44.904412 24442 raft_consensus.cc:2454] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e04f9d945c14636bc08ef0f3183fa79 in term 4.
I20260807 11:08:44.905244 24744 consensus_queue.cc:237] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:44.915946 24078 catalog_manager.cc:5495] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 reported cstate change: term changed from 3 to 4, leader changed from 212ebf83bd07438a82bd50130d0f427d (127.23.120.131) to 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130). New cstate: current_term: 4 leader_uuid: "2e04f9d945c14636bc08ef0f3183fa79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: UNKNOWN } } }
I20260807 11:08:44.974174 24738 raft_consensus.cc:979] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a: : Instructing follower 2e04f9d945c14636bc08ef0f3183fa79 to start an election
I20260807 11:08:44.974886 24743 raft_consensus.cc:1067] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [term 2 LEADER]: Signalling peer 2e04f9d945c14636bc08ef0f3183fa79 to start an election
I20260807 11:08:44.976886 24333 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1308b5fb7ff942a69ffe1f28a83642bb"
dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
from {username='slave'} at 127.0.0.1:53624
I20260807 11:08:44.977617 24333 raft_consensus.cc:482] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:08:44.978118 24333 raft_consensus.cc:3037] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:08:44.985633 24333 raft_consensus.cc:504] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:44.988545 24333 leader_election.cc:288] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 3 election: Requested vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:44.989610 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1308b5fb7ff942a69ffe1f28a83642bb" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
I20260807 11:08:44.990563 24228 raft_consensus.cc:3032] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:08:44.990409 24442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1308b5fb7ff942a69ffe1f28a83642bb" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
I20260807 11:08:44.991195 24228 raft_consensus.cc:729] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 625cae382881486e90d3a22d03b7ef7a, State: Running, Role: LEADER
I20260807 11:08:44.991442 24442 raft_consensus.cc:3037] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:08:44.992543 24228 consensus_queue.cc:260] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:44.994901 24228 raft_consensus.cc:3037] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:08:45.003080 24442 raft_consensus.cc:2454] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e04f9d945c14636bc08ef0f3183fa79 in term 3.
I20260807 11:08:45.005069 24285 leader_election.cc:302] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d, 2e04f9d945c14636bc08ef0f3183fa79; no voters:
I20260807 11:08:45.006224 24228 raft_consensus.cc:2454] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e04f9d945c14636bc08ef0f3183fa79 in term 3.
I20260807 11:08:45.006459 24744 raft_consensus.cc:2781] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:08:45.007136 24744 raft_consensus.cc:686] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 LEADER]: Becoming Leader. State: Replica: 2e04f9d945c14636bc08ef0f3183fa79, State: Running, Role: LEADER
I20260807 11:08:45.008615 24744 consensus_queue.cc:237] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:45.018774 24078 catalog_manager.cc:5495] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79 reported cstate change: term changed from 2 to 3, leader changed from 625cae382881486e90d3a22d03b7ef7a (127.23.120.129) to 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130). New cstate: current_term: 3 leader_uuid: "2e04f9d945c14636bc08ef0f3183fa79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: UNKNOWN } } }
I20260807 11:08:45.095994 24738 raft_consensus.cc:979] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a: : Instructing follower 2e04f9d945c14636bc08ef0f3183fa79 to start an election
I20260807 11:08:45.096532 24743 raft_consensus.cc:1067] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [term 2 LEADER]: Signalling peer 2e04f9d945c14636bc08ef0f3183fa79 to start an election
I20260807 11:08:45.098140 24333 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9f8dec87030d45c9a4b51dd3842292d1"
dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
from {username='slave'} at 127.0.0.1:53624
I20260807 11:08:45.098806 24333 raft_consensus.cc:482] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:08:45.099260 24333 raft_consensus.cc:3037] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:08:45.106374 24333 raft_consensus.cc:504] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:45.109041 24333 leader_election.cc:288] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 3 election: Requested vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:45.109803 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8dec87030d45c9a4b51dd3842292d1" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
I20260807 11:08:45.110594 24228 raft_consensus.cc:3032] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:08:45.110544 24442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8dec87030d45c9a4b51dd3842292d1" candidate_uuid: "2e04f9d945c14636bc08ef0f3183fa79" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
I20260807 11:08:45.110993 24228 raft_consensus.cc:729] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 625cae382881486e90d3a22d03b7ef7a, State: Running, Role: LEADER
I20260807 11:08:45.111305 24442 raft_consensus.cc:3037] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:08:45.111918 24228 consensus_queue.cc:260] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:45.113334 24228 raft_consensus.cc:3037] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:08:45.117395 24442 raft_consensus.cc:2454] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e04f9d945c14636bc08ef0f3183fa79 in term 3.
I20260807 11:08:45.118726 24285 leader_election.cc:302] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d, 2e04f9d945c14636bc08ef0f3183fa79; no voters:
I20260807 11:08:45.120028 24744 raft_consensus.cc:2781] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:08:45.120939 24744 raft_consensus.cc:686] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 LEADER]: Becoming Leader. State: Replica: 2e04f9d945c14636bc08ef0f3183fa79, State: Running, Role: LEADER
I20260807 11:08:45.121675 24228 raft_consensus.cc:2454] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e04f9d945c14636bc08ef0f3183fa79 in term 3.
I20260807 11:08:45.122277 24744 consensus_queue.cc:237] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:45.133424 24078 catalog_manager.cc:5495] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 reported cstate change: term changed from 2 to 3, leader changed from 625cae382881486e90d3a22d03b7ef7a (127.23.120.129) to 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130). New cstate: current_term: 3 leader_uuid: "2e04f9d945c14636bc08ef0f3183fa79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: UNKNOWN } } }
I20260807 11:08:45.304430 24228 raft_consensus.cc:1261] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Refusing update from remote peer 2e04f9d945c14636bc08ef0f3183fa79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:08:45.306499 24744 consensus_queue.cc:1035] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:08:45.323861 24442 raft_consensus.cc:1261] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Refusing update from remote peer 2e04f9d945c14636bc08ef0f3183fa79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:08:45.326380 24742 consensus_queue.cc:1035] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:08:45.383924 24228 raft_consensus.cc:1261] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 4 FOLLOWER]: Refusing update from remote peer 2e04f9d945c14636bc08ef0f3183fa79: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 11:08:45.385530 24767 consensus_queue.cc:1035] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:08:45.399451 24442 raft_consensus.cc:1261] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 4 FOLLOWER]: Refusing update from remote peer 2e04f9d945c14636bc08ef0f3183fa79: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 11:08:45.400907 24767 consensus_queue.cc:1035] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:08:45.439467 24443 raft_consensus.cc:1261] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [term 3 FOLLOWER]: Refusing update from remote peer 2e04f9d945c14636bc08ef0f3183fa79: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:08:45.440912 24742 consensus_queue.cc:1035] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:08:45.458905 24221 raft_consensus.cc:1261] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [term 4 FOLLOWER]: Refusing update from remote peer 2e04f9d945c14636bc08ef0f3183fa79: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 11:08:45.459326 24228 raft_consensus.cc:1261] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [term 3 FOLLOWER]: Refusing update from remote peer 2e04f9d945c14636bc08ef0f3183fa79: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:08:45.460806 24742 consensus_queue.cc:1035] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:08:45.464740 24775 consensus_queue.cc:1035] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260807 11:08:45.485361 24443 raft_consensus.cc:1261] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [term 4 FOLLOWER]: Refusing update from remote peer 2e04f9d945c14636bc08ef0f3183fa79: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 11:08:45.487002 24742 consensus_queue.cc:1035] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:08:45.555420 24443 raft_consensus.cc:1261] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [term 3 FOLLOWER]: Refusing update from remote peer 2e04f9d945c14636bc08ef0f3183fa79: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:08:45.556984 24775 consensus_queue.cc:1035] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:08:45.574491 24228 raft_consensus.cc:1261] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [term 3 FOLLOWER]: Refusing update from remote peer 2e04f9d945c14636bc08ef0f3183fa79: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:08:45.576411 24775 consensus_queue.cc:1035] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:08:47.594789 24034 auto_leader_rebalancer-test.cc:303] The leader distribution is
I20260807 11:08:47.595513 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:08:47.602574 24333 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1d0a8dfa345949b192adaddd29002763"
dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
mode: GRACEFUL
new_leader_uuid: "625cae382881486e90d3a22d03b7ef7a"
from {username='slave'} at 127.0.0.1:53646
I20260807 11:08:47.603175 24333 raft_consensus.cc:595] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 LEADER]: Received request to transfer leadership to TS 625cae382881486e90d3a22d03b7ef7a
I20260807 11:08:47.605808 24333 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5a1d2a0a8e38476ea55fdd85717cf7ea"
dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
mode: GRACEFUL
new_leader_uuid: "625cae382881486e90d3a22d03b7ef7a"
from {username='slave'} at 127.0.0.1:53646
I20260807 11:08:47.606282 24333 raft_consensus.cc:595] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 LEADER]: Received request to transfer leadership to TS 625cae382881486e90d3a22d03b7ef7a
I20260807 11:08:47.608284 24333 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9dff94141ffa4c789f8e39fcdfe20b9b"
dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
mode: GRACEFUL
new_leader_uuid: "212ebf83bd07438a82bd50130d0f427d"
from {username='slave'} at 127.0.0.1:53646
I20260807 11:08:47.608718 24333 raft_consensus.cc:595] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 LEADER]: Received request to transfer leadership to TS 212ebf83bd07438a82bd50130d0f427d
I20260807 11:08:47.610699 24333 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9f8dec87030d45c9a4b51dd3842292d1"
dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
mode: GRACEFUL
new_leader_uuid: "625cae382881486e90d3a22d03b7ef7a"
from {username='slave'} at 127.0.0.1:53646
I20260807 11:08:47.611137 24333 raft_consensus.cc:595] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 LEADER]: Received request to transfer leadership to TS 625cae382881486e90d3a22d03b7ef7a
I20260807 11:08:47.613077 24333 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b0993a417f1b4e32ab72ff3d30fbacef"
dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
mode: GRACEFUL
new_leader_uuid: "212ebf83bd07438a82bd50130d0f427d"
from {username='slave'} at 127.0.0.1:53646
I20260807 11:08:47.613502 24333 raft_consensus.cc:595] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 4 LEADER]: Received request to transfer leadership to TS 212ebf83bd07438a82bd50130d0f427d
I20260807 11:08:47.614442 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260807 11:08:47.614885 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:08:47.821545 24786 raft_consensus.cc:979] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79: : Instructing follower 212ebf83bd07438a82bd50130d0f427d to start an election
I20260807 11:08:47.822085 24790 raft_consensus.cc:1067] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 4 LEADER]: Signalling peer 212ebf83bd07438a82bd50130d0f427d to start an election
I20260807 11:08:47.823557 24442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b0993a417f1b4e32ab72ff3d30fbacef"
dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
from {username='slave'} at 127.0.0.1:41062
I20260807 11:08:47.824024 24442 raft_consensus.cc:482] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:08:47.824301 24442 raft_consensus.cc:3037] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:08:47.828408 24442 raft_consensus.cc:504] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:47.830302 24442 leader_election.cc:288] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 5 election: Requested vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129)
I20260807 11:08:47.831122 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0993a417f1b4e32ab72ff3d30fbacef" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
I20260807 11:08:47.831506 24333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0993a417f1b4e32ab72ff3d30fbacef" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
I20260807 11:08:47.831681 24228 raft_consensus.cc:3037] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:08:47.832053 24333 raft_consensus.cc:3032] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:08:47.832345 24333 raft_consensus.cc:729] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2e04f9d945c14636bc08ef0f3183fa79, State: Running, Role: LEADER
I20260807 11:08:47.833021 24333 consensus_queue.cc:260] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:47.833969 24333 raft_consensus.cc:3037] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:08:47.836318 24228 raft_consensus.cc:2454] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212ebf83bd07438a82bd50130d0f427d in term 5.
I20260807 11:08:47.837579 24393 leader_election.cc:302] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d, 625cae382881486e90d3a22d03b7ef7a; no voters:
I20260807 11:08:47.838332 24777 raft_consensus.cc:2781] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:08:47.838747 24333 raft_consensus.cc:2454] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212ebf83bd07438a82bd50130d0f427d in term 5.
I20260807 11:08:47.839557 24777 raft_consensus.cc:686] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 5 LEADER]: Becoming Leader. State: Replica: 212ebf83bd07438a82bd50130d0f427d, State: Running, Role: LEADER
I20260807 11:08:47.840509 24777 consensus_queue.cc:237] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:47.849021 24080 catalog_manager.cc:5495] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d reported cstate change: term changed from 4 to 5, leader changed from 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130) to 212ebf83bd07438a82bd50130d0f427d (127.23.120.131). New cstate: current_term: 5 leader_uuid: "212ebf83bd07438a82bd50130d0f427d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: HEALTHY } } }
I20260807 11:08:47.864462 24786 raft_consensus.cc:979] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79: : Instructing follower 625cae382881486e90d3a22d03b7ef7a to start an election
I20260807 11:08:47.864964 24790 raft_consensus.cc:1067] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 LEADER]: Signalling peer 625cae382881486e90d3a22d03b7ef7a to start an election
I20260807 11:08:47.866331 24228 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1d0a8dfa345949b192adaddd29002763"
dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
from {username='slave'} at 127.0.0.1:39162
I20260807 11:08:47.866715 24228 raft_consensus.cc:482] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:08:47.866950 24228 raft_consensus.cc:3037] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:08:47.870957 24228 raft_consensus.cc:504] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:47.872625 24228 leader_election.cc:288] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 3 election: Requested vote from peers 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:47.873502 24333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0a8dfa345949b192adaddd29002763" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
I20260807 11:08:47.873819 24442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0a8dfa345949b192adaddd29002763" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
I20260807 11:08:47.874119 24333 raft_consensus.cc:3032] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:08:47.874364 24442 raft_consensus.cc:3037] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:08:47.874487 24333 raft_consensus.cc:729] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2e04f9d945c14636bc08ef0f3183fa79, State: Running, Role: LEADER
I20260807 11:08:47.875270 24333 consensus_queue.cc:260] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:47.876552 24333 raft_consensus.cc:3037] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:08:47.880971 24442 raft_consensus.cc:2454] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625cae382881486e90d3a22d03b7ef7a in term 3.
I20260807 11:08:47.882247 24174 leader_election.cc:302] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d, 625cae382881486e90d3a22d03b7ef7a; no voters:
I20260807 11:08:47.883183 24333 raft_consensus.cc:2454] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625cae382881486e90d3a22d03b7ef7a in term 3.
I20260807 11:08:47.884035 24799 raft_consensus.cc:2781] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:08:47.885303 24799 raft_consensus.cc:686] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a [term 3 LEADER]: Becoming Leader. State: Replica: 625cae382881486e90d3a22d03b7ef7a, State: Running, Role: LEADER
I20260807 11:08:47.886108 24799 consensus_queue.cc:237] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:47.892609 24080 catalog_manager.cc:5495] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a reported cstate change: term changed from 2 to 3, leader changed from 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130) to 625cae382881486e90d3a22d03b7ef7a (127.23.120.129). New cstate: current_term: 3 leader_uuid: "625cae382881486e90d3a22d03b7ef7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: UNKNOWN } } }
I20260807 11:08:47.908941 24786 raft_consensus.cc:979] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79: : Instructing follower 625cae382881486e90d3a22d03b7ef7a to start an election
I20260807 11:08:47.909456 24790 raft_consensus.cc:1067] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 LEADER]: Signalling peer 625cae382881486e90d3a22d03b7ef7a to start an election
I20260807 11:08:47.910796 24228 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5a1d2a0a8e38476ea55fdd85717cf7ea"
dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
from {username='slave'} at 127.0.0.1:39162
I20260807 11:08:47.911304 24228 raft_consensus.cc:482] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:08:47.911589 24228 raft_consensus.cc:3037] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:08:47.915608 24228 raft_consensus.cc:504] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:47.917037 24228 leader_election.cc:288] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 3 election: Requested vote from peers 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:47.917892 24333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1d2a0a8e38476ea55fdd85717cf7ea" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
I20260807 11:08:47.918144 24442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1d2a0a8e38476ea55fdd85717cf7ea" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
I20260807 11:08:47.918414 24333 raft_consensus.cc:3032] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:08:47.918713 24442 raft_consensus.cc:3037] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:08:47.918802 24333 raft_consensus.cc:729] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2e04f9d945c14636bc08ef0f3183fa79, State: Running, Role: LEADER
I20260807 11:08:47.919768 24333 consensus_queue.cc:260] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:47.920955 24333 raft_consensus.cc:3037] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:08:47.923487 24442 raft_consensus.cc:2454] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 212ebf83bd07438a82bd50130d0f427d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625cae382881486e90d3a22d03b7ef7a in term 3.
I20260807 11:08:47.924515 24174 leader_election.cc:302] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d, 625cae382881486e90d3a22d03b7ef7a; no voters:
I20260807 11:08:47.925289 24799 raft_consensus.cc:2781] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:08:47.925757 24799 raft_consensus.cc:686] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [term 3 LEADER]: Becoming Leader. State: Replica: 625cae382881486e90d3a22d03b7ef7a, State: Running, Role: LEADER
I20260807 11:08:47.925758 24333 raft_consensus.cc:2454] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625cae382881486e90d3a22d03b7ef7a in term 3.
I20260807 11:08:47.926764 24799 consensus_queue.cc:237] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:47.934674 24080 catalog_manager.cc:5495] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a reported cstate change: term changed from 2 to 3, leader changed from 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130) to 625cae382881486e90d3a22d03b7ef7a (127.23.120.129). New cstate: current_term: 3 leader_uuid: "625cae382881486e90d3a22d03b7ef7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: UNKNOWN } } }
I20260807 11:08:48.020270 24786 raft_consensus.cc:979] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79: : Instructing follower 625cae382881486e90d3a22d03b7ef7a to start an election
I20260807 11:08:48.020895 24790 raft_consensus.cc:1067] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 LEADER]: Signalling peer 625cae382881486e90d3a22d03b7ef7a to start an election
I20260807 11:08:48.022343 24228 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9f8dec87030d45c9a4b51dd3842292d1"
dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
from {username='slave'} at 127.0.0.1:39162
I20260807 11:08:48.022817 24228 raft_consensus.cc:482] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:08:48.023093 24228 raft_consensus.cc:3037] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:08:48.027659 24228 raft_consensus.cc:504] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:48.029325 24228 leader_election.cc:288] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 4 election: Requested vote from peers 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129), 212ebf83bd07438a82bd50130d0f427d (127.23.120.131:43609)
I20260807 11:08:48.030216 24333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8dec87030d45c9a4b51dd3842292d1" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
I20260807 11:08:48.030428 24442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8dec87030d45c9a4b51dd3842292d1" candidate_uuid: "625cae382881486e90d3a22d03b7ef7a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
I20260807 11:08:48.030874 24333 raft_consensus.cc:3032] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:08:48.030920 24442 raft_consensus.cc:3037] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:08:48.031335 24333 raft_consensus.cc:729] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2e04f9d945c14636bc08ef0f3183fa79, State: Running, Role: LEADER
I20260807 11:08:48.032186 24333 consensus_queue.cc:260] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:48.033505 24333 raft_consensus.cc:3037] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:08:48.037612 24442 raft_consensus.cc:2454] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625cae382881486e90d3a22d03b7ef7a in term 4.
I20260807 11:08:48.038617 24174 leader_election.cc:302] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d, 625cae382881486e90d3a22d03b7ef7a; no voters:
I20260807 11:08:48.039440 24799 raft_consensus.cc:2781] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:08:48.039896 24799 raft_consensus.cc:686] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [term 4 LEADER]: Becoming Leader. State: Replica: 625cae382881486e90d3a22d03b7ef7a, State: Running, Role: LEADER
I20260807 11:08:48.040153 24333 raft_consensus.cc:2454] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625cae382881486e90d3a22d03b7ef7a in term 4.
I20260807 11:08:48.040863 24799 consensus_queue.cc:237] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:48.049777 24786 raft_consensus.cc:979] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79: : Instructing follower 212ebf83bd07438a82bd50130d0f427d to start an election
I20260807 11:08:48.050257 24790 raft_consensus.cc:1067] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 LEADER]: Signalling peer 212ebf83bd07438a82bd50130d0f427d to start an election
I20260807 11:08:48.049319 24080 catalog_manager.cc:5495] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a reported cstate change: term changed from 3 to 4, leader changed from 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130) to 625cae382881486e90d3a22d03b7ef7a (127.23.120.129). New cstate: current_term: 4 leader_uuid: "625cae382881486e90d3a22d03b7ef7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: UNKNOWN } } }
I20260807 11:08:48.051793 24442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9dff94141ffa4c789f8e39fcdfe20b9b"
dest_uuid: "212ebf83bd07438a82bd50130d0f427d"
from {username='slave'} at 127.0.0.1:41062
I20260807 11:08:48.052292 24442 raft_consensus.cc:482] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:08:48.052606 24442 raft_consensus.cc:3037] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:48.057713 24442 raft_consensus.cc:504] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:48.059911 24442 leader_election.cc:288] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 2 election: Requested vote from peers 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841), 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129)
I20260807 11:08:48.060513 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dff94141ffa4c789f8e39fcdfe20b9b" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "625cae382881486e90d3a22d03b7ef7a"
I20260807 11:08:48.061075 24228 raft_consensus.cc:3037] T 9dff94141ffa4c789f8e39fcdfe20b9b P 625cae382881486e90d3a22d03b7ef7a [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:48.061081 24333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dff94141ffa4c789f8e39fcdfe20b9b" candidate_uuid: "212ebf83bd07438a82bd50130d0f427d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2e04f9d945c14636bc08ef0f3183fa79"
I20260807 11:08:48.061730 24333 raft_consensus.cc:3032] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:08:48.061980 24333 raft_consensus.cc:729] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2e04f9d945c14636bc08ef0f3183fa79, State: Running, Role: LEADER
I20260807 11:08:48.062764 24333 consensus_queue.cc:260] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:48.063745 24333 raft_consensus.cc:3037] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:08:48.065425 24228 raft_consensus.cc:2454] T 9dff94141ffa4c789f8e39fcdfe20b9b P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212ebf83bd07438a82bd50130d0f427d in term 2.
I20260807 11:08:48.066278 24393 leader_election.cc:302] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212ebf83bd07438a82bd50130d0f427d, 625cae382881486e90d3a22d03b7ef7a; no voters:
I20260807 11:08:48.066879 24777 raft_consensus.cc:2781] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:08:48.067396 24777 raft_consensus.cc:686] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d [term 2 LEADER]: Becoming Leader. State: Replica: 212ebf83bd07438a82bd50130d0f427d, State: Running, Role: LEADER
I20260807 11:08:48.068322 24777 consensus_queue.cc:237] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } }
I20260807 11:08:48.070361 24333 raft_consensus.cc:2454] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212ebf83bd07438a82bd50130d0f427d in term 2.
I20260807 11:08:48.077304 24077 catalog_manager.cc:5495] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d reported cstate change: term changed from 1 to 2, leader changed from 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130) to 212ebf83bd07438a82bd50130d0f427d (127.23.120.131). New cstate: current_term: 2 leader_uuid: "212ebf83bd07438a82bd50130d0f427d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 } health_report { overall_health: HEALTHY } } }
I20260807 11:08:48.347225 24228 raft_consensus.cc:1261] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 5 FOLLOWER]: Refusing update from remote peer 212ebf83bd07438a82bd50130d0f427d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260807 11:08:48.348382 24777 consensus_queue.cc:1035] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [LEADER]: Connected to new peer: Peer: permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 11:08:48.358603 24333 raft_consensus.cc:1261] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 5 FOLLOWER]: Refusing update from remote peer 212ebf83bd07438a82bd50130d0f427d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260807 11:08:48.359938 24791 consensus_queue.cc:1035] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 11:08:48.382810 24442 raft_consensus.cc:1261] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [term 3 FOLLOWER]: Refusing update from remote peer 625cae382881486e90d3a22d03b7ef7a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:08:48.383946 24799 consensus_queue.cc:1035] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:08:48.392724 24333 raft_consensus.cc:1261] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 FOLLOWER]: Refusing update from remote peer 625cae382881486e90d3a22d03b7ef7a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:08:48.394119 24822 consensus_queue.cc:1035] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:08:48.453042 24333 raft_consensus.cc:1261] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 FOLLOWER]: Refusing update from remote peer 625cae382881486e90d3a22d03b7ef7a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:08:48.454288 24822 consensus_queue.cc:1035] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:08:48.462440 24442 raft_consensus.cc:1261] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 212ebf83bd07438a82bd50130d0f427d [term 3 FOLLOWER]: Refusing update from remote peer 625cae382881486e90d3a22d03b7ef7a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:08:48.463864 24799 consensus_queue.cc:1035] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:08:48.553107 24333 raft_consensus.cc:1261] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [term 4 FOLLOWER]: Refusing update from remote peer 625cae382881486e90d3a22d03b7ef7a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 11:08:48.554332 24799 consensus_queue.cc:1035] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:08:48.560551 24333 raft_consensus.cc:1261] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Refusing update from remote peer 212ebf83bd07438a82bd50130d0f427d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:08:48.561987 24791 consensus_queue.cc:1035] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e04f9d945c14636bc08ef0f3183fa79" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 39129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:08:48.562410 24442 raft_consensus.cc:1261] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [term 4 FOLLOWER]: Refusing update from remote peer 625cae382881486e90d3a22d03b7ef7a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 11:08:48.563747 24799 consensus_queue.cc:1035] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "212ebf83bd07438a82bd50130d0f427d" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 43609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:08:48.572458 24228 raft_consensus.cc:1261] T 9dff94141ffa4c789f8e39fcdfe20b9b P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Refusing update from remote peer 212ebf83bd07438a82bd50130d0f427d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:08:48.573865 24835 consensus_queue.cc:1035] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d [LEADER]: Connected to new peer: Peer: permanent_uuid: "625cae382881486e90d3a22d03b7ef7a" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 38841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:08:48.615626 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:08:48.620561 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:08:48.620939 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:08:49.621707 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:08:49.627138 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:08:49.627585 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:08:50.628398 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:08:50.633092 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:08:50.633508 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:08:51.634331 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:08:51.639808 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:08:51.640269 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:08:52.641171 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:08:52.647671 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:08:52.648294 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:08:53.649398 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:08:53.654806 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:08:53.655231 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:08:54.656123 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:08:54.660796 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:08:54.661206 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:08:55.662007 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:08:55.667276 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:08:55.667685 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:08:56.668431 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:08:56.673318 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:08:56.673770 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:08:57.708206 24034 auto_leader_rebalancer-test.cc:314] The leader distribution is
I20260807 11:08:57.709084 24034 tablet_server.cc:178] TabletServer@127.23.120.129:0 shutting down...
I20260807 11:08:57.754289 24034 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260807 11:08:57.755103 24034 tablet_replica.cc:330] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a: stopping tablet replica
I20260807 11:08:57.755952 24034 raft_consensus.cc:2229] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a [term 3 LEADER]: Raft consensus shutting down.
I20260807 11:08:57.757053 24034 raft_consensus.cc:2258] T 1d0a8dfa345949b192adaddd29002763 P 625cae382881486e90d3a22d03b7ef7a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 11:08:57.759363 24034 tablet_replica.cc:330] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a: stopping tablet replica
I20260807 11:08:57.759886 24034 raft_consensus.cc:2229] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 11:08:57.760351 24034 raft_consensus.cc:2258] T d9426ebc1a024747ad29dd72e9a763d4 P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 11:08:57.762025 24034 tablet_replica.cc:330] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a: stopping tablet replica
I20260807 11:08:57.762529 24034 raft_consensus.cc:2229] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [term 4 LEADER]: Raft consensus shutting down.
I20260807 11:08:57.763262 24034 raft_consensus.cc:2258] T 9f8dec87030d45c9a4b51dd3842292d1 P 625cae382881486e90d3a22d03b7ef7a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 11:08:57.764846 24034 tablet_replica.cc:330] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a: stopping tablet replica
I20260807 11:08:57.765355 24034 raft_consensus.cc:2229] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260807 11:08:57.765832 24034 raft_consensus.cc:2258] T 143ad113a591461abce196dc159fa9f7 P 625cae382881486e90d3a22d03b7ef7a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 11:08:57.767412 24034 tablet_replica.cc:330] T 9dff94141ffa4c789f8e39fcdfe20b9b P 625cae382881486e90d3a22d03b7ef7a: stopping tablet replica
I20260807 11:08:57.767879 24034 raft_consensus.cc:2229] T 9dff94141ffa4c789f8e39fcdfe20b9b P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 11:08:57.768285 24034 raft_consensus.cc:2258] T 9dff94141ffa4c789f8e39fcdfe20b9b P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 11:08:57.769840 24034 tablet_replica.cc:330] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a: stopping tablet replica
I20260807 11:08:57.770293 24034 raft_consensus.cc:2229] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 11:08:57.770709 24034 raft_consensus.cc:2258] T bc59e5e4c27b459e938d8942f3ff65c3 P 625cae382881486e90d3a22d03b7ef7a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 11:08:57.772154 24034 tablet_replica.cc:330] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a: stopping tablet replica
I20260807 11:08:57.772712 24034 raft_consensus.cc:2229] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 5 FOLLOWER]: Raft consensus shutting down.
I20260807 11:08:57.773366 24034 raft_consensus.cc:2258] T b0993a417f1b4e32ab72ff3d30fbacef P 625cae382881486e90d3a22d03b7ef7a [term 5 FOLLOWER]: Raft consensus is shut down!
I20260807 11:08:57.775565 24034 tablet_replica.cc:330] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a: stopping tablet replica
I20260807 11:08:57.776086 24034 raft_consensus.cc:2229] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [term 3 LEADER]: Raft consensus shutting down.
I20260807 11:08:57.776814 24034 raft_consensus.cc:2258] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 625cae382881486e90d3a22d03b7ef7a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 11:08:57.778230 24034 tablet_replica.cc:330] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a: stopping tablet replica
I20260807 11:08:57.778611 24034 raft_consensus.cc:2229] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 11:08:57.778949 24034 raft_consensus.cc:2258] T 1308b5fb7ff942a69ffe1f28a83642bb P 625cae382881486e90d3a22d03b7ef7a [term 3 FOLLOWER]: Raft consensus is shut down!
W20260807 11:08:57.885946 24283 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.120.129:38841: connect: Connection refused (error 111)
W20260807 11:08:57.901368 24283 consensus_peers.cc:487] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79 -> Peer 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841): Couldn't send request to peer 625cae382881486e90d3a22d03b7ef7a. Status: Network error: Client connection negotiation failed: client connection to 127.23.120.129:38841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 11:08:57.901904 24034 tablet_server.cc:194] TabletServer@127.23.120.129:0 shutdown complete.
I20260807 11:08:57.921522 24034 tablet_server.cc:178] TabletServer@127.23.120.130:0 shutting down...
W20260807 11:08:57.932050 24393 consensus_peers.cc:487] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d -> Peer 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841): Couldn't send request to peer 625cae382881486e90d3a22d03b7ef7a. Status: Network error: Client connection negotiation failed: client connection to 127.23.120.129:38841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 11:08:57.953735 24034 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260807 11:08:57.954345 24034 tablet_replica.cc:330] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79: stopping tablet replica
I20260807 11:08:57.954996 24034 raft_consensus.cc:2229] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 LEADER]: Raft consensus shutting down.
I20260807 11:08:57.955962 24034 raft_consensus.cc:2258] T 1308b5fb7ff942a69ffe1f28a83642bb P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 11:08:57.958132 24034 tablet_replica.cc:330] T bc59e5e4c27b459e938d8942f3ff65c3 P 2e04f9d945c14636bc08ef0f3183fa79: stopping tablet replica
I20260807 11:08:57.958956 24034 raft_consensus.cc:2229] T bc59e5e4c27b459e938d8942f3ff65c3 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 11:08:57.959462 24034 raft_consensus.cc:2258] T bc59e5e4c27b459e938d8942f3ff65c3 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260807 11:08:57.960280 24283 consensus_peers.cc:487] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 -> Peer 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841): Couldn't send request to peer 625cae382881486e90d3a22d03b7ef7a. Status: Network error: Client connection negotiation failed: client connection to 127.23.120.129:38841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 11:08:57.961364 24034 tablet_replica.cc:330] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79: stopping tablet replica
I20260807 11:08:57.961942 24034 raft_consensus.cc:2229] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 11:08:57.962412 24034 raft_consensus.cc:2258] T 9dff94141ffa4c789f8e39fcdfe20b9b P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 11:08:57.964085 24034 tablet_replica.cc:330] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79: stopping tablet replica
I20260807 11:08:57.964591 24034 raft_consensus.cc:2229] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 LEADER]: Raft consensus shutting down.
I20260807 11:08:57.965415 24034 raft_consensus.cc:2258] T d9426ebc1a024747ad29dd72e9a763d4 P 2e04f9d945c14636bc08ef0f3183fa79 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 11:08:57.966980 24034 tablet_replica.cc:330] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79: stopping tablet replica
I20260807 11:08:57.967483 24034 raft_consensus.cc:2229] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [term 4 LEADER]: Raft consensus shutting down.
I20260807 11:08:57.968268 24034 raft_consensus.cc:2258] T 143ad113a591461abce196dc159fa9f7 P 2e04f9d945c14636bc08ef0f3183fa79 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 11:08:57.970046 24034 tablet_replica.cc:330] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79: stopping tablet replica
I20260807 11:08:57.970564 24034 raft_consensus.cc:2229] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260807 11:08:57.971011 24034 raft_consensus.cc:2258] T b0993a417f1b4e32ab72ff3d30fbacef P 2e04f9d945c14636bc08ef0f3183fa79 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260807 11:08:57.972736 24034 tablet_replica.cc:330] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79: stopping tablet replica
I20260807 11:08:57.973297 24034 raft_consensus.cc:2229] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 11:08:57.973745 24034 raft_consensus.cc:2258] T 1d0a8dfa345949b192adaddd29002763 P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 11:08:57.975603 24034 tablet_replica.cc:330] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79: stopping tablet replica
I20260807 11:08:57.976121 24034 raft_consensus.cc:2229] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260807 11:08:57.976648 24034 raft_consensus.cc:2258] T 9f8dec87030d45c9a4b51dd3842292d1 P 2e04f9d945c14636bc08ef0f3183fa79 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 11:08:57.978570 24034 tablet_replica.cc:330] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79: stopping tablet replica
I20260807 11:08:57.979053 24034 raft_consensus.cc:2229] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 11:08:57.979516 24034 raft_consensus.cc:2258] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 2e04f9d945c14636bc08ef0f3183fa79 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260807 11:08:57.980050 24399 consensus_peers.cc:487] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d -> Peer 2e04f9d945c14636bc08ef0f3183fa79 (127.23.120.130:39129): Couldn't send request to peer 2e04f9d945c14636bc08ef0f3183fa79. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:08:58.008251 24034 tablet_server.cc:194] TabletServer@127.23.120.130:0 shutdown complete.
I20260807 11:08:58.028394 24034 tablet_server.cc:178] TabletServer@127.23.120.131:0 shutting down...
I20260807 11:08:58.057533 24034 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260807 11:08:58.058408 24034 tablet_replica.cc:330] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d: stopping tablet replica
I20260807 11:08:58.059530 24034 raft_consensus.cc:2229] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [term 4 FOLLOWER]: Raft consensus shutting down.
I20260807 11:08:58.060047 24034 raft_consensus.cc:2258] T 143ad113a591461abce196dc159fa9f7 P 212ebf83bd07438a82bd50130d0f427d [term 4 FOLLOWER]: Raft consensus is shut down!
W20260807 11:08:58.062129 24393 consensus_peers.cc:487] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d -> Peer 625cae382881486e90d3a22d03b7ef7a (127.23.120.129:38841): Couldn't send request to peer 625cae382881486e90d3a22d03b7ef7a. Status: Network error: Client connection negotiation failed: client connection to 127.23.120.129:38841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 11:08:58.062216 24034 tablet_replica.cc:330] T d9426ebc1a024747ad29dd72e9a763d4 P 212ebf83bd07438a82bd50130d0f427d: stopping tablet replica
I20260807 11:08:58.062824 24034 raft_consensus.cc:2229] T d9426ebc1a024747ad29dd72e9a763d4 P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 11:08:58.063294 24034 raft_consensus.cc:2258] T d9426ebc1a024747ad29dd72e9a763d4 P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 11:08:58.065094 24034 tablet_replica.cc:330] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d: stopping tablet replica
I20260807 11:08:58.065596 24034 raft_consensus.cc:2229] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 11:08:58.066036 24034 raft_consensus.cc:2258] T 1308b5fb7ff942a69ffe1f28a83642bb P 212ebf83bd07438a82bd50130d0f427d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 11:08:58.067714 24034 tablet_replica.cc:330] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d: stopping tablet replica
I20260807 11:08:58.068233 24034 raft_consensus.cc:2229] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 5 LEADER]: Raft consensus shutting down.
I20260807 11:08:58.069114 24034 raft_consensus.cc:2258] T b0993a417f1b4e32ab72ff3d30fbacef P 212ebf83bd07438a82bd50130d0f427d [term 5 FOLLOWER]: Raft consensus is shut down!
I20260807 11:08:58.070672 24034 tablet_replica.cc:330] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 212ebf83bd07438a82bd50130d0f427d: stopping tablet replica
I20260807 11:08:58.071184 24034 raft_consensus.cc:2229] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 212ebf83bd07438a82bd50130d0f427d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 11:08:58.071605 24034 raft_consensus.cc:2258] T 5a1d2a0a8e38476ea55fdd85717cf7ea P 212ebf83bd07438a82bd50130d0f427d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 11:08:58.073397 24034 tablet_replica.cc:330] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d: stopping tablet replica
I20260807 11:08:58.073946 24034 raft_consensus.cc:2229] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [term 4 FOLLOWER]: Raft consensus shutting down.
I20260807 11:08:58.074435 24034 raft_consensus.cc:2258] T 9f8dec87030d45c9a4b51dd3842292d1 P 212ebf83bd07438a82bd50130d0f427d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 11:08:58.076113 24034 tablet_replica.cc:330] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d: stopping tablet replica
I20260807 11:08:58.076597 24034 raft_consensus.cc:2229] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 11:08:58.077098 24034 raft_consensus.cc:2258] T 1d0a8dfa345949b192adaddd29002763 P 212ebf83bd07438a82bd50130d0f427d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 11:08:58.078950 24034 tablet_replica.cc:330] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d: stopping tablet replica
I20260807 11:08:58.079442 24034 raft_consensus.cc:2229] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d [term 2 LEADER]: Raft consensus shutting down.
I20260807 11:08:58.080168 24034 raft_consensus.cc:2258] T 9dff94141ffa4c789f8e39fcdfe20b9b P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 11:08:58.082008 24034 tablet_replica.cc:330] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d: stopping tablet replica
I20260807 11:08:58.082516 24034 raft_consensus.cc:2229] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d [term 2 LEADER]: Raft consensus shutting down.
I20260807 11:08:58.083293 24034 raft_consensus.cc:2258] T bc59e5e4c27b459e938d8942f3ff65c3 P 212ebf83bd07438a82bd50130d0f427d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 11:08:58.112665 24034 tablet_server.cc:194] TabletServer@127.23.120.131:0 shutdown complete.
I20260807 11:08:58.130288 24034 master.cc:537] Master@127.23.120.190:46829 shutting down...
I20260807 11:08:58.239425 24034 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0244322652a34ae4b3b7e3b01e7ec2b2 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:08:58.240032 24034 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0244322652a34ae4b3b7e3b01e7ec2b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:08:58.240410 24034 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0244322652a34ae4b3b7e3b01e7ec2b2: stopping tablet replica
I20260807 11:08:58.283241 24034 master.cc:552] Master@127.23.120.190:46829 shutdown complete.
212ebf83bd07438a82bd50130d0f427d 1
2e04f9d945c14636bc08ef0f3183fa79 4
625cae382881486e90d3a22d03b7ef7a 4
212ebf83bd07438a82bd50130d0f427d 3
2e04f9d945c14636bc08ef0f3183fa79 3
625cae382881486e90d3a22d03b7ef7a 3
212ebf83bd07438a82bd50130d0f427d 1
2e04f9d945c14636bc08ef0f3183fa79 8
625cae382881486e90d3a22d03b7ef7a 0
212ebf83bd07438a82bd50130d0f427d 3
2e04f9d945c14636bc08ef0f3183fa79 3
625cae382881486e90d3a22d03b7ef7a 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (35736 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260807 11:08:58.331221 24034 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.120.190:46611
I20260807 11:08:58.332365 24034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:08:58.338804 24856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:08:58.339643 24860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:08:58.340081 24858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:08:58.340477 24034 server_base.cc:1034] running on GCE node
I20260807 11:08:58.342043 24034 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 11:08:58.342265 24034 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 11:08:58.342476 24034 hybrid_clock.cc:648] HybridClock initialized: now 1786100938342443 us; error 0 us; skew 500 ppm
I20260807 11:08:58.343034 24034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:08:58.345600 24034 webserver.cc:460] Webserver started at http://127.23.120.190:44291/ using document root <none> and password file <none>
I20260807 11:08:58.346114 24034 fs_manager.cc:359] Metadata directory not provided
I20260807 11:08:58.346319 24034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:08:58.346623 24034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:08:58.347853 24034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786100902301536-24034-0/minicluster-data/master-0-root/instance:
uuid: "2e2233d81e9e472a90a0fb6f42174060"
format_stamp: "Formatted at 2026-08-07 11:08:58 on dist-test-slave-g170"
I20260807 11:08:58.352507 24034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 11:08:58.356365 24873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:08:58.357098 24034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 11:08:58.357376 24034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786100902301536-24034-0/minicluster-data/master-0-root
uuid: "2e2233d81e9e472a90a0fb6f42174060"
format_stamp: "Formatted at 2026-08-07 11:08:58 on dist-test-slave-g170"
I20260807 11:08:58.357652 24034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786100902301536-24034-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786100902301536-24034-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786100902301536-24034-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:08:58.393250 24034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:08:58.394497 24034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:08:58.433053 24034 rpc_server.cc:307] RPC server started. Bound to: 127.23.120.190:46611
I20260807 11:08:58.433157 24942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.120.190:46611 every 8 connection(s)
I20260807 11:08:58.436703 24944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:58.446799 24944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e2233d81e9e472a90a0fb6f42174060: Bootstrap starting.
I20260807 11:08:58.451413 24944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e2233d81e9e472a90a0fb6f42174060: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:58.455565 24944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e2233d81e9e472a90a0fb6f42174060: No bootstrap required, opened a new log
I20260807 11:08:58.457612 24944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e2233d81e9e472a90a0fb6f42174060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2233d81e9e472a90a0fb6f42174060" member_type: VOTER }
I20260807 11:08:58.458112 24944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e2233d81e9e472a90a0fb6f42174060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:58.458390 24944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e2233d81e9e472a90a0fb6f42174060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2233d81e9e472a90a0fb6f42174060, State: Initialized, Role: FOLLOWER
I20260807 11:08:58.458987 24944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e2233d81e9e472a90a0fb6f42174060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2233d81e9e472a90a0fb6f42174060" member_type: VOTER }
I20260807 11:08:58.459451 24944 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e2233d81e9e472a90a0fb6f42174060 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:08:58.459715 24944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e2233d81e9e472a90a0fb6f42174060 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:08:58.460009 24944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e2233d81e9e472a90a0fb6f42174060 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:58.463708 24944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e2233d81e9e472a90a0fb6f42174060 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2233d81e9e472a90a0fb6f42174060" member_type: VOTER }
I20260807 11:08:58.464192 24944 leader_election.cc:302] T 00000000000000000000000000000000 P 2e2233d81e9e472a90a0fb6f42174060 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e2233d81e9e472a90a0fb6f42174060; no voters:
I20260807 11:08:58.465327 24944 leader_election.cc:288] T 00000000000000000000000000000000 P 2e2233d81e9e472a90a0fb6f42174060 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:08:58.465723 24948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e2233d81e9e472a90a0fb6f42174060 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:08:58.467023 24948 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e2233d81e9e472a90a0fb6f42174060 [term 1 LEADER]: Becoming Leader. State: Replica: 2e2233d81e9e472a90a0fb6f42174060, State: Running, Role: LEADER
I20260807 11:08:58.467684 24948 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e2233d81e9e472a90a0fb6f42174060 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2233d81e9e472a90a0fb6f42174060" member_type: VOTER }
I20260807 11:08:58.468281 24944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e2233d81e9e472a90a0fb6f42174060 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:08:58.473263 24950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e2233d81e9e472a90a0fb6f42174060 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e2233d81e9e472a90a0fb6f42174060. Latest consensus state: current_term: 1 leader_uuid: "2e2233d81e9e472a90a0fb6f42174060" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2233d81e9e472a90a0fb6f42174060" member_type: VOTER } }
I20260807 11:08:58.473969 24950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e2233d81e9e472a90a0fb6f42174060 [sys.catalog]: This master's current role is: LEADER
I20260807 11:08:58.474359 24949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e2233d81e9e472a90a0fb6f42174060 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e2233d81e9e472a90a0fb6f42174060" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2233d81e9e472a90a0fb6f42174060" member_type: VOTER } }
I20260807 11:08:58.474797 24949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e2233d81e9e472a90a0fb6f42174060 [sys.catalog]: This master's current role is: LEADER
I20260807 11:08:58.476086 24961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:08:58.481110 24961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:08:58.484470 24034 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260807 11:08:58.489691 24961 catalog_manager.cc:1347] Generated new cluster ID: 6cf104216dc5452b921be4a41655fe9b
I20260807 11:08:58.489925 24961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:08:58.517508 24961 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:08:58.519137 24961 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:08:58.532182 24961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e2233d81e9e472a90a0fb6f42174060: Generated new TSK 0
I20260807 11:08:58.532788 24961 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:08:58.551198 24034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:08:58.557035 24978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:08:58.559087 24979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:08:58.560494 24981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:08:58.561300 24034 server_base.cc:1034] running on GCE node
I20260807 11:08:58.562309 24034 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 11:08:58.562482 24034 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 11:08:58.562597 24034 hybrid_clock.cc:648] HybridClock initialized: now 1786100938562587 us; error 0 us; skew 500 ppm
I20260807 11:08:58.563042 24034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:08:58.565277 24034 webserver.cc:460] Webserver started at http://127.23.120.129:33285/ using document root <none> and password file <none>
I20260807 11:08:58.565724 24034 fs_manager.cc:359] Metadata directory not provided
I20260807 11:08:58.565881 24034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:08:58.566102 24034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:08:58.567097 24034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786100902301536-24034-0/minicluster-data/ts-0-root/instance:
uuid: "720dbff207a94212bdaf227d9b05d627"
format_stamp: "Formatted at 2026-08-07 11:08:58 on dist-test-slave-g170"
I20260807 11:08:58.571489 24034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260807 11:08:58.574536 24987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:08:58.575241 24034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260807 11:08:58.575481 24034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786100902301536-24034-0/minicluster-data/ts-0-root
uuid: "720dbff207a94212bdaf227d9b05d627"
format_stamp: "Formatted at 2026-08-07 11:08:58 on dist-test-slave-g170"
I20260807 11:08:58.575711 24034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786100902301536-24034-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786100902301536-24034-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786100902301536-24034-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:08:58.589272 24034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:08:58.590351 24034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:08:58.591671 24034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:08:58.593863 24034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:08:58.594053 24034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:08:58.594313 24034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:08:58.594484 24034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:08:58.634433 24034 rpc_server.cc:307] RPC server started. Bound to: 127.23.120.129:37989
I20260807 11:08:58.634546 25076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.120.129:37989 every 8 connection(s)
I20260807 11:08:58.639281 24034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:08:58.646922 25081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:08:58.648890 25082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:08:58.651944 25084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:08:58.652872 24034 server_base.cc:1034] running on GCE node
I20260807 11:08:58.654623 24034 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 11:08:58.654840 24034 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 11:08:58.655036 24034 hybrid_clock.cc:648] HybridClock initialized: now 1786100938655019 us; error 0 us; skew 500 ppm
I20260807 11:08:58.655105 25077 heartbeater.cc:347] Connected to a master server at 127.23.120.190:46611
I20260807 11:08:58.655521 25077 heartbeater.cc:464] Registering TS with master...
I20260807 11:08:58.655661 24034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:08:58.656301 25077 heartbeater.cc:511] Master 127.23.120.190:46611 requested a full tablet report, sending...
I20260807 11:08:58.658732 24900 ts_manager.cc:194] Registered new tserver with Master: 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989)
I20260807 11:08:58.658895 24034 webserver.cc:460] Webserver started at http://127.23.120.130:37335/ using document root <none> and password file <none>
I20260807 11:08:58.659493 24034 fs_manager.cc:359] Metadata directory not provided
I20260807 11:08:58.659713 24034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:08:58.659991 24034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:08:58.660921 24900 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51230
I20260807 11:08:58.661329 24034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786100902301536-24034-0/minicluster-data/ts-1-root/instance:
uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
format_stamp: "Formatted at 2026-08-07 11:08:58 on dist-test-slave-g170"
I20260807 11:08:58.665884 24034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 11:08:58.669201 25089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:08:58.670024 24034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260807 11:08:58.670300 24034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786100902301536-24034-0/minicluster-data/ts-1-root
uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
format_stamp: "Formatted at 2026-08-07 11:08:58 on dist-test-slave-g170"
I20260807 11:08:58.670585 24034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786100902301536-24034-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786100902301536-24034-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786100902301536-24034-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:08:58.683585 24034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:08:58.684816 24034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:08:58.686332 24034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:08:58.688510 24034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:08:58.688723 24034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:08:58.689010 24034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:08:58.689167 24034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:08:58.729481 24034 rpc_server.cc:307] RPC server started. Bound to: 127.23.120.130:44315
I20260807 11:08:58.729598 25187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.120.130:44315 every 8 connection(s)
I20260807 11:08:58.734251 24034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:08:58.741955 25193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:08:58.744226 25196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:08:58.745327 25199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:08:58.746055 24034 server_base.cc:1034] running on GCE node
I20260807 11:08:58.747042 24034 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 11:08:58.747329 24034 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 11:08:58.747515 24034 hybrid_clock.cc:648] HybridClock initialized: now 1786100938747500 us; error 0 us; skew 500 ppm
I20260807 11:08:58.748193 24034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:08:58.749271 25188 heartbeater.cc:347] Connected to a master server at 127.23.120.190:46611
I20260807 11:08:58.749701 25188 heartbeater.cc:464] Registering TS with master...
I20260807 11:08:58.750345 25188 heartbeater.cc:511] Master 127.23.120.190:46611 requested a full tablet report, sending...
I20260807 11:08:58.750921 24034 webserver.cc:460] Webserver started at http://127.23.120.131:44037/ using document root <none> and password file <none>
I20260807 11:08:58.751526 24034 fs_manager.cc:359] Metadata directory not provided
I20260807 11:08:58.751730 24034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:08:58.752002 24034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:08:58.752384 24900 ts_manager.cc:194] Registered new tserver with Master: bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315)
I20260807 11:08:58.753240 24034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786100902301536-24034-0/minicluster-data/ts-2-root/instance:
uuid: "3751e3c65b0340eeba13019e16fcc3bf"
format_stamp: "Formatted at 2026-08-07 11:08:58 on dist-test-slave-g170"
I20260807 11:08:58.754005 24900 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51234
I20260807 11:08:58.757771 24034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260807 11:08:58.761659 25206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:08:58.762707 24034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:08:58.763006 24034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786100902301536-24034-0/minicluster-data/ts-2-root
uuid: "3751e3c65b0340eeba13019e16fcc3bf"
format_stamp: "Formatted at 2026-08-07 11:08:58 on dist-test-slave-g170"
I20260807 11:08:58.763299 24034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786100902301536-24034-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786100902301536-24034-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786100902301536-24034-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:08:58.804672 24034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:08:58.805934 24034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:08:58.807389 24034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:08:58.809538 24034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:08:58.809763 24034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:08:58.810009 24034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:08:58.810184 24034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:08:58.851370 24034 rpc_server.cc:307] RPC server started. Bound to: 127.23.120.131:45885
I20260807 11:08:58.851503 25296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.120.131:45885 every 8 connection(s)
I20260807 11:08:58.867884 25297 heartbeater.cc:347] Connected to a master server at 127.23.120.190:46611
I20260807 11:08:58.868284 25297 heartbeater.cc:464] Registering TS with master...
I20260807 11:08:58.869063 25297 heartbeater.cc:511] Master 127.23.120.190:46611 requested a full tablet report, sending...
I20260807 11:08:58.870970 24900 ts_manager.cc:194] Registered new tserver with Master: 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:08:58.871743 24034 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017093108s
I20260807 11:08:58.873064 24900 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51242
I20260807 11:08:58.874264 24034 test_util.cc:268] Using random seed: 134126980
I20260807 11:08:58.898339 24900 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51248:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:08:58.901278 24900 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:08:58.988313 25249 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4964373b984d6089c2c83d13a52792 (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260807 11:08:58.988596 25245 tablet_service.cc:1467] Processing CreateTablet for tablet 1924e8081c31446c8953300614a93d43 (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260807 11:08:58.989795 25249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4964373b984d6089c2c83d13a52792. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:58.990444 25245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1924e8081c31446c8953300614a93d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:58.993762 25137 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4964373b984d6089c2c83d13a52792 (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260807 11:08:58.993893 25126 tablet_service.cc:1467] Processing CreateTablet for tablet a59d5a742d82444391f7103f701851bf (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260807 11:08:58.995267 25137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4964373b984d6089c2c83d13a52792. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:58.995898 25126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a59d5a742d82444391f7103f701851bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:58.997491 25133 tablet_service.cc:1467] Processing CreateTablet for tablet 1924e8081c31446c8953300614a93d43 (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260807 11:08:58.998174 25242 tablet_service.cc:1467] Processing CreateTablet for tablet 89018784373e4903ae1a1bcaa1ac3f4e (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260807 11:08:58.998900 25133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1924e8081c31446c8953300614a93d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:58.999543 25242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89018784373e4903ae1a1bcaa1ac3f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:59.001976 25132 tablet_service.cc:1467] Processing CreateTablet for tablet 89018784373e4903ae1a1bcaa1ac3f4e (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260807 11:08:59.003322 25132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89018784373e4903ae1a1bcaa1ac3f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:59.006021 25130 tablet_service.cc:1467] Processing CreateTablet for tablet 425296c8f30a46e0a77af3752db337bf (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260807 11:08:59.006464 25240 tablet_service.cc:1467] Processing CreateTablet for tablet 425296c8f30a46e0a77af3752db337bf (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260807 11:08:59.007366 25130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 425296c8f30a46e0a77af3752db337bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:59.007817 25240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 425296c8f30a46e0a77af3752db337bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:59.009969 25129 tablet_service.cc:1467] Processing CreateTablet for tablet fd44192986994fc086a65a73dea598d6 (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260807 11:08:59.011298 25129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd44192986994fc086a65a73dea598d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:59.013976 25127 tablet_service.cc:1467] Processing CreateTablet for tablet c1eee668b4cc4eddad42edaf1827309d (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260807 11:08:59.015283 25127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1eee668b4cc4eddad42edaf1827309d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:59.014935 25239 tablet_service.cc:1467] Processing CreateTablet for tablet fd44192986994fc086a65a73dea598d6 (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260807 11:08:59.016647 25239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd44192986994fc086a65a73dea598d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:59.022578 25123 tablet_service.cc:1467] Processing CreateTablet for tablet 39c732ca6d3a44758b2f1f46d56f9161 (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260807 11:08:59.024408 25123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c732ca6d3a44758b2f1f46d56f9161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:59.026713 25119 tablet_service.cc:1467] Processing CreateTablet for tablet dfd144f74da942029b3904cc9cfda2a2 (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260807 11:08:59.028224 25119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfd144f74da942029b3904cc9cfda2a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:59.029326 25238 tablet_service.cc:1467] Processing CreateTablet for tablet c1eee668b4cc4eddad42edaf1827309d (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260807 11:08:59.031250 25238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1eee668b4cc4eddad42edaf1827309d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:59.034128 25118 tablet_service.cc:1467] Processing CreateTablet for tablet 5e25a8abd35e45c2af9bdf5f6b053269 (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260807 11:08:59.035493 25118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e25a8abd35e45c2af9bdf5f6b053269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:59.047883 25237 tablet_service.cc:1467] Processing CreateTablet for tablet a59d5a742d82444391f7103f701851bf (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260807 11:08:59.049211 25237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a59d5a742d82444391f7103f701851bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:59.058466 25236 tablet_service.cc:1467] Processing CreateTablet for tablet 39c732ca6d3a44758b2f1f46d56f9161 (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260807 11:08:59.062340 25236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c732ca6d3a44758b2f1f46d56f9161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:59.076509 25234 tablet_service.cc:1467] Processing CreateTablet for tablet dfd144f74da942029b3904cc9cfda2a2 (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260807 11:08:59.078099 25234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfd144f74da942029b3904cc9cfda2a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:59.089983 25233 tablet_service.cc:1467] Processing CreateTablet for tablet 5e25a8abd35e45c2af9bdf5f6b053269 (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260807 11:08:59.091677 25233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e25a8abd35e45c2af9bdf5f6b053269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:59.101660 25035 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4964373b984d6089c2c83d13a52792 (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260807 11:08:59.101933 25031 tablet_service.cc:1467] Processing CreateTablet for tablet 1924e8081c31446c8953300614a93d43 (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260807 11:08:59.102981 25035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4964373b984d6089c2c83d13a52792. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:59.104048 25031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1924e8081c31446c8953300614a93d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:59.113976 25027 tablet_service.cc:1467] Processing CreateTablet for tablet 89018784373e4903ae1a1bcaa1ac3f4e (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260807 11:08:59.115343 25027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89018784373e4903ae1a1bcaa1ac3f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:59.122378 25026 tablet_service.cc:1467] Processing CreateTablet for tablet 425296c8f30a46e0a77af3752db337bf (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260807 11:08:59.123669 25026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 425296c8f30a46e0a77af3752db337bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:59.130496 25025 tablet_service.cc:1467] Processing CreateTablet for tablet fd44192986994fc086a65a73dea598d6 (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260807 11:08:59.132045 25025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd44192986994fc086a65a73dea598d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:59.138617 25023 tablet_service.cc:1467] Processing CreateTablet for tablet c1eee668b4cc4eddad42edaf1827309d (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260807 11:08:59.140355 25023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1eee668b4cc4eddad42edaf1827309d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:59.142249 25022 tablet_service.cc:1467] Processing CreateTablet for tablet a59d5a742d82444391f7103f701851bf (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260807 11:08:59.144480 25021 tablet_service.cc:1467] Processing CreateTablet for tablet 39c732ca6d3a44758b2f1f46d56f9161 (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260807 11:08:59.149197 25021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c732ca6d3a44758b2f1f46d56f9161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:59.151203 25018 tablet_service.cc:1467] Processing CreateTablet for tablet 5e25a8abd35e45c2af9bdf5f6b053269 (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260807 11:08:59.153076 25018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e25a8abd35e45c2af9bdf5f6b053269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:59.149192 25020 tablet_service.cc:1467] Processing CreateTablet for tablet dfd144f74da942029b3904cc9cfda2a2 (DEFAULT_TABLE table=test-workload [id=e9c8be7d0fbd443ca8f60e7e9892d516]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260807 11:08:59.154471 25020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfd144f74da942029b3904cc9cfda2a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:59.157166 25022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a59d5a742d82444391f7103f701851bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:08:59.190649 25325 tablet_bootstrap.cc:492] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627: Bootstrap starting.
I20260807 11:08:59.198727 25325 tablet_bootstrap.cc:654] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.208385 25323 tablet_bootstrap.cc:492] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf: Bootstrap starting.
I20260807 11:08:59.231595 25323 tablet_bootstrap.cc:654] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.212541 25324 tablet_bootstrap.cc:492] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861: Bootstrap starting.
I20260807 11:08:59.243599 25324 tablet_bootstrap.cc:654] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.244107 25325 tablet_bootstrap.cc:492] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627: No bootstrap required, opened a new log
I20260807 11:08:59.244416 25323 tablet_bootstrap.cc:492] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf: No bootstrap required, opened a new log
I20260807 11:08:59.244652 25325 ts_tablet_manager.cc:1397] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627: Time spent bootstrapping tablet: real 0.054s user 0.013s sys 0.004s
I20260807 11:08:59.244918 25323 ts_tablet_manager.cc:1397] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.009s
I20260807 11:08:59.248157 25324 tablet_bootstrap.cc:492] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861: No bootstrap required, opened a new log
I20260807 11:08:59.247764 25325 raft_consensus.cc:348] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } } peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } }
I20260807 11:08:59.247934 25323 raft_consensus.cc:348] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } } peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } }
I20260807 11:08:59.248683 25324 ts_tablet_manager.cc:1397] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861: Time spent bootstrapping tablet: real 0.046s user 0.008s sys 0.004s
I20260807 11:08:59.248710 25325 raft_consensus.cc:374] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.248952 25323 raft_consensus.cc:374] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.249287 25325 raft_consensus.cc:729] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720dbff207a94212bdaf227d9b05d627, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.249339 25323 raft_consensus.cc:729] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3751e3c65b0340eeba13019e16fcc3bf, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.250240 25323 consensus_queue.cc:260] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } } peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } }
I20260807 11:08:59.250195 25325 consensus_queue.cc:260] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } } peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } }
I20260807 11:08:59.251101 25324 raft_consensus.cc:348] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.252351 25324 raft_consensus.cc:374] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.252629 25324 raft_consensus.cc:729] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbeb19c5189a46bcb4c327db2b6bc861, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.253391 25324 consensus_queue.cc:260] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.256265 25325 ts_tablet_manager.cc:1428] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260807 11:08:59.256515 25077 heartbeater.cc:503] Master 127.23.120.190:46611 was elected leader, sending a full tablet report...
I20260807 11:08:59.256899 25323 ts_tablet_manager.cc:1428] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf: Time spent starting tablet: real 0.012s user 0.005s sys 0.007s
I20260807 11:08:59.257304 25325 tablet_bootstrap.cc:492] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627: Bootstrap starting.
I20260807 11:08:59.257952 25323 tablet_bootstrap.cc:492] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf: Bootstrap starting.
I20260807 11:08:59.261358 25297 heartbeater.cc:503] Master 127.23.120.190:46611 was elected leader, sending a full tablet report...
I20260807 11:08:59.263022 25324 ts_tablet_manager.cc:1428] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861: Time spent starting tablet: real 0.014s user 0.011s sys 0.000s
I20260807 11:08:59.263729 25188 heartbeater.cc:503] Master 127.23.120.190:46611 was elected leader, sending a full tablet report...
I20260807 11:08:59.264074 25325 tablet_bootstrap.cc:654] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.271263 25323 tablet_bootstrap.cc:654] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.273566 25324 tablet_bootstrap.cc:492] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861: Bootstrap starting.
I20260807 11:08:59.280562 25324 tablet_bootstrap.cc:654] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.286218 25323 tablet_bootstrap.cc:492] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf: No bootstrap required, opened a new log
I20260807 11:08:59.286739 25323 ts_tablet_manager.cc:1397] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.018s
I20260807 11:08:59.287231 25325 tablet_bootstrap.cc:492] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627: No bootstrap required, opened a new log
I20260807 11:08:59.287789 25325 ts_tablet_manager.cc:1397] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.016s
I20260807 11:08:59.289963 25323 raft_consensus.cc:348] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.290812 25323 raft_consensus.cc:374] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.290668 25325 raft_consensus.cc:348] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.291157 25323 raft_consensus.cc:729] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3751e3c65b0340eeba13019e16fcc3bf, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.291523 25325 raft_consensus.cc:374] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.291846 25325 raft_consensus.cc:729] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720dbff207a94212bdaf227d9b05d627, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.292071 25323 consensus_queue.cc:260] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.292588 25325 consensus_queue.cc:260] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.294548 25323 ts_tablet_manager.cc:1428] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf: Time spent starting tablet: real 0.007s user 0.000s sys 0.008s
I20260807 11:08:59.295506 25323 tablet_bootstrap.cc:492] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf: Bootstrap starting.
I20260807 11:08:59.296751 25325 ts_tablet_manager.cc:1428] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20260807 11:08:59.297909 25325 tablet_bootstrap.cc:492] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627: Bootstrap starting.
I20260807 11:08:59.302239 25323 tablet_bootstrap.cc:654] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.304414 25325 tablet_bootstrap.cc:654] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.305239 25324 tablet_bootstrap.cc:492] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861: No bootstrap required, opened a new log
I20260807 11:08:59.305703 25324 ts_tablet_manager.cc:1397] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.002s
I20260807 11:08:59.308449 25324 raft_consensus.cc:348] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.309176 25324 raft_consensus.cc:374] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.309481 25324 raft_consensus.cc:729] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbeb19c5189a46bcb4c327db2b6bc861, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.310150 25334 raft_consensus.cc:482] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:59.310396 25324 consensus_queue.cc:260] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.310595 25334 raft_consensus.cc:504] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } } peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } }
I20260807 11:08:59.312839 25324 ts_tablet_manager.cc:1428] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20260807 11:08:59.314381 25324 tablet_bootstrap.cc:492] T dfd144f74da942029b3904cc9cfda2a2 P bbeb19c5189a46bcb4c327db2b6bc861: Bootstrap starting.
I20260807 11:08:59.320358 25324 tablet_bootstrap.cc:654] T dfd144f74da942029b3904cc9cfda2a2 P bbeb19c5189a46bcb4c327db2b6bc861: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.326128 25334 leader_election.cc:288] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:08:59.340948 25323 tablet_bootstrap.cc:492] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf: No bootstrap required, opened a new log
I20260807 11:08:59.343906 25153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4964373b984d6089c2c83d13a52792" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" is_pre_election: true
I20260807 11:08:59.344731 25336 raft_consensus.cc:482] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:59.345178 25323 ts_tablet_manager.cc:1397] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf: Time spent bootstrapping tablet: real 0.050s user 0.029s sys 0.013s
I20260807 11:08:59.345289 25336 raft_consensus.cc:504] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
W20260807 11:08:59.346277 24990 leader_election.cc:341] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:59.347409 25323 raft_consensus.cc:348] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.348075 25336 leader_election.cc:288] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:08:59.348208 25323 raft_consensus.cc:374] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.348611 25323 raft_consensus.cc:729] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3751e3c65b0340eeba13019e16fcc3bf, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.350557 25323 consensus_queue.cc:260] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.354941 25324 tablet_bootstrap.cc:492] T dfd144f74da942029b3904cc9cfda2a2 P bbeb19c5189a46bcb4c327db2b6bc861: No bootstrap required, opened a new log
I20260807 11:08:59.355535 25324 ts_tablet_manager.cc:1397] T dfd144f74da942029b3904cc9cfda2a2 P bbeb19c5189a46bcb4c327db2b6bc861: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.002s
W20260807 11:08:59.355545 25299 tablet.cc:2369] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:08:59.358057 25323 ts_tablet_manager.cc:1428] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260807 11:08:59.358937 25324 raft_consensus.cc:348] T dfd144f74da942029b3904cc9cfda2a2 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.359697 25323 tablet_bootstrap.cc:492] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf: Bootstrap starting.
I20260807 11:08:59.359858 25324 raft_consensus.cc:374] T dfd144f74da942029b3904cc9cfda2a2 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.360355 25324 raft_consensus.cc:729] T dfd144f74da942029b3904cc9cfda2a2 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbeb19c5189a46bcb4c327db2b6bc861, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.361318 25324 consensus_queue.cc:260] T dfd144f74da942029b3904cc9cfda2a2 P bbeb19c5189a46bcb4c327db2b6bc861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.364267 25324 ts_tablet_manager.cc:1428] T dfd144f74da942029b3904cc9cfda2a2 P bbeb19c5189a46bcb4c327db2b6bc861: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260807 11:08:59.365374 25324 tablet_bootstrap.cc:492] T 7e4964373b984d6089c2c83d13a52792 P bbeb19c5189a46bcb4c327db2b6bc861: Bootstrap starting.
I20260807 11:08:59.365669 25323 tablet_bootstrap.cc:654] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.370594 25325 tablet_bootstrap.cc:492] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627: No bootstrap required, opened a new log
I20260807 11:08:59.371197 25325 ts_tablet_manager.cc:1397] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627: Time spent bootstrapping tablet: real 0.073s user 0.014s sys 0.034s
I20260807 11:08:59.373780 25324 tablet_bootstrap.cc:654] T 7e4964373b984d6089c2c83d13a52792 P bbeb19c5189a46bcb4c327db2b6bc861: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.374902 25325 raft_consensus.cc:348] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.376125 25325 raft_consensus.cc:374] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.376499 25325 raft_consensus.cc:729] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720dbff207a94212bdaf227d9b05d627, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.381606 25324 tablet_bootstrap.cc:492] T 7e4964373b984d6089c2c83d13a52792 P bbeb19c5189a46bcb4c327db2b6bc861: No bootstrap required, opened a new log
I20260807 11:08:59.382125 25324 ts_tablet_manager.cc:1397] T 7e4964373b984d6089c2c83d13a52792 P bbeb19c5189a46bcb4c327db2b6bc861: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.002s
I20260807 11:08:59.381421 25325 consensus_queue.cc:260] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.384922 25324 raft_consensus.cc:348] T 7e4964373b984d6089c2c83d13a52792 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } } peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } }
I20260807 11:08:59.385708 25324 raft_consensus.cc:374] T 7e4964373b984d6089c2c83d13a52792 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.386008 25324 raft_consensus.cc:729] T 7e4964373b984d6089c2c83d13a52792 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbeb19c5189a46bcb4c327db2b6bc861, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.386690 25325 ts_tablet_manager.cc:1428] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627: Time spent starting tablet: real 0.015s user 0.005s sys 0.003s
I20260807 11:08:59.386704 25324 consensus_queue.cc:260] T 7e4964373b984d6089c2c83d13a52792 P bbeb19c5189a46bcb4c327db2b6bc861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } } peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } }
I20260807 11:08:59.388031 25325 tablet_bootstrap.cc:492] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627: Bootstrap starting.
I20260807 11:08:59.389854 25324 ts_tablet_manager.cc:1428] T 7e4964373b984d6089c2c83d13a52792 P bbeb19c5189a46bcb4c327db2b6bc861: Time spent starting tablet: real 0.007s user 0.002s sys 0.006s
I20260807 11:08:59.393282 25324 tablet_bootstrap.cc:492] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861: Bootstrap starting.
I20260807 11:08:59.400929 25323 tablet_bootstrap.cc:492] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf: No bootstrap required, opened a new log
I20260807 11:08:59.401438 25323 ts_tablet_manager.cc:1397] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf: Time spent bootstrapping tablet: real 0.043s user 0.012s sys 0.022s
I20260807 11:08:59.401273 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4964373b984d6089c2c83d13a52792" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf" is_pre_election: true
I20260807 11:08:59.401960 25261 raft_consensus.cc:2454] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 720dbff207a94212bdaf227d9b05d627 in term 0.
I20260807 11:08:59.403278 24993 leader_election.cc:302] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, 720dbff207a94212bdaf227d9b05d627; no voters: bbeb19c5189a46bcb4c327db2b6bc861
I20260807 11:08:59.404099 25323 raft_consensus.cc:348] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.404851 25323 raft_consensus.cc:374] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.405023 25334 raft_consensus.cc:2781] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:08:59.405176 25323 raft_consensus.cc:729] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3751e3c65b0340eeba13019e16fcc3bf, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.405421 25334 raft_consensus.cc:482] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:08:59.405776 25334 raft_consensus.cc:3037] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:59.405961 25323 consensus_queue.cc:260] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.407015 25325 tablet_bootstrap.cc:654] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.408330 25323 ts_tablet_manager.cc:1428] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260807 11:08:59.416678 25323 tablet_bootstrap.cc:492] T 39c732ca6d3a44758b2f1f46d56f9161 P 3751e3c65b0340eeba13019e16fcc3bf: Bootstrap starting.
I20260807 11:08:59.415663 25334 raft_consensus.cc:504] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } } peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } }
I20260807 11:08:59.419813 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e25a8abd35e45c2af9bdf5f6b053269" candidate_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf" is_pre_election: true
I20260807 11:08:59.420621 25334 leader_election.cc:288] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 1 election: Requested vote from peers bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:08:59.420701 25261 raft_consensus.cc:2454] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbeb19c5189a46bcb4c327db2b6bc861 in term 0.
I20260807 11:08:59.422281 25095 leader_election.cc:302] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, bbeb19c5189a46bcb4c327db2b6bc861; no voters:
I20260807 11:08:59.422482 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4964373b984d6089c2c83d13a52792" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
I20260807 11:08:59.423087 25261 raft_consensus.cc:3037] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:59.423329 25153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4964373b984d6089c2c83d13a52792" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
I20260807 11:08:59.424160 25336 raft_consensus.cc:2781] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:08:59.424526 25336 raft_consensus.cc:482] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:08:59.425496 25336 raft_consensus.cc:3037] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:59.425136 25153 raft_consensus.cc:3037] T 7e4964373b984d6089c2c83d13a52792 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:59.429785 25323 tablet_bootstrap.cc:654] T 39c732ca6d3a44758b2f1f46d56f9161 P 3751e3c65b0340eeba13019e16fcc3bf: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.426062 25048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e25a8abd35e45c2af9bdf5f6b053269" candidate_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720dbff207a94212bdaf227d9b05d627" is_pre_election: true
I20260807 11:08:59.432826 25261 raft_consensus.cc:2454] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720dbff207a94212bdaf227d9b05d627 in term 1.
I20260807 11:08:59.427079 25324 tablet_bootstrap.cc:654] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861: Neither blocks nor log segments found. Creating new log.
W20260807 11:08:59.432983 25096 leader_election.cc:341] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:59.434878 25325 tablet_bootstrap.cc:492] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627: No bootstrap required, opened a new log
I20260807 11:08:59.434880 24993 leader_election.cc:302] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, 720dbff207a94212bdaf227d9b05d627; no voters:
I20260807 11:08:59.438198 25334 raft_consensus.cc:2781] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:08:59.438474 25153 raft_consensus.cc:2454] T 7e4964373b984d6089c2c83d13a52792 P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720dbff207a94212bdaf227d9b05d627 in term 1.
I20260807 11:08:59.439491 25325 ts_tablet_manager.cc:1397] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.013s
I20260807 11:08:59.432454 25336 raft_consensus.cc:504] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.448151 25048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e25a8abd35e45c2af9bdf5f6b053269" candidate_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720dbff207a94212bdaf227d9b05d627"
I20260807 11:08:59.448660 25336 leader_election.cc:288] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 1 election: Requested vote from peers 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
W20260807 11:08:59.450618 25096 leader_election.cc:341] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:59.448472 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e25a8abd35e45c2af9bdf5f6b053269" candidate_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
I20260807 11:08:59.447932 25325 raft_consensus.cc:348] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.451610 25261 raft_consensus.cc:3037] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:59.451586 25325 raft_consensus.cc:374] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.452740 25325 raft_consensus.cc:729] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720dbff207a94212bdaf227d9b05d627, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.452270 25334 raft_consensus.cc:686] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [term 1 LEADER]: Becoming Leader. State: Replica: 720dbff207a94212bdaf227d9b05d627, State: Running, Role: LEADER
I20260807 11:08:59.453605 25325 consensus_queue.cc:260] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.454489 25323 tablet_bootstrap.cc:492] T 39c732ca6d3a44758b2f1f46d56f9161 P 3751e3c65b0340eeba13019e16fcc3bf: No bootstrap required, opened a new log
I20260807 11:08:59.453997 25334 consensus_queue.cc:237] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } } peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } }
I20260807 11:08:59.454937 25323 ts_tablet_manager.cc:1397] T 39c732ca6d3a44758b2f1f46d56f9161 P 3751e3c65b0340eeba13019e16fcc3bf: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.009s
I20260807 11:08:59.456439 25325 ts_tablet_manager.cc:1428] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627: Time spent starting tablet: real 0.017s user 0.002s sys 0.005s
I20260807 11:08:59.457526 25325 tablet_bootstrap.cc:492] T dfd144f74da942029b3904cc9cfda2a2 P 720dbff207a94212bdaf227d9b05d627: Bootstrap starting.
I20260807 11:08:59.457602 25323 raft_consensus.cc:348] T 39c732ca6d3a44758b2f1f46d56f9161 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.458393 25323 raft_consensus.cc:374] T 39c732ca6d3a44758b2f1f46d56f9161 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.458688 25323 raft_consensus.cc:729] T 39c732ca6d3a44758b2f1f46d56f9161 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3751e3c65b0340eeba13019e16fcc3bf, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.459367 25323 consensus_queue.cc:260] T 39c732ca6d3a44758b2f1f46d56f9161 P 3751e3c65b0340eeba13019e16fcc3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.460047 25261 raft_consensus.cc:2454] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbeb19c5189a46bcb4c327db2b6bc861 in term 1.
I20260807 11:08:59.460566 25324 tablet_bootstrap.cc:492] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861: No bootstrap required, opened a new log
I20260807 11:08:59.463285 25324 ts_tablet_manager.cc:1397] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861: Time spent bootstrapping tablet: real 0.071s user 0.012s sys 0.001s
I20260807 11:08:59.463774 25095 leader_election.cc:302] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, bbeb19c5189a46bcb4c327db2b6bc861; no voters: 720dbff207a94212bdaf227d9b05d627
I20260807 11:08:59.464226 25325 tablet_bootstrap.cc:654] T dfd144f74da942029b3904cc9cfda2a2 P 720dbff207a94212bdaf227d9b05d627: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.465229 25336 raft_consensus.cc:2781] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:08:59.466344 25324 raft_consensus.cc:348] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.467579 25324 raft_consensus.cc:374] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.461951 25323 ts_tablet_manager.cc:1428] T 39c732ca6d3a44758b2f1f46d56f9161 P 3751e3c65b0340eeba13019e16fcc3bf: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20260807 11:08:59.473058 25362 raft_consensus.cc:482] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:59.473568 25362 raft_consensus.cc:504] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.475973 25362 leader_election.cc:288] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:08:59.478657 25368 raft_consensus.cc:482] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:59.468034 25324 raft_consensus.cc:729] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbeb19c5189a46bcb4c327db2b6bc861, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.476820 25153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89018784373e4903ae1a1bcaa1ac3f4e" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" is_pre_election: true
W20260807 11:08:59.479624 24966 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 11:08:59.479856 25324 consensus_queue.cc:260] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
W20260807 11:08:59.480695 24990 leader_election.cc:341] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:59.481556 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89018784373e4903ae1a1bcaa1ac3f4e" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf" is_pre_election: true
I20260807 11:08:59.477315 25336 raft_consensus.cc:686] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 LEADER]: Becoming Leader. State: Replica: bbeb19c5189a46bcb4c327db2b6bc861, State: Running, Role: LEADER
I20260807 11:08:59.482393 25324 ts_tablet_manager.cc:1428] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260807 11:08:59.483369 25324 tablet_bootstrap.cc:492] T fd44192986994fc086a65a73dea598d6 P bbeb19c5189a46bcb4c327db2b6bc861: Bootstrap starting.
I20260807 11:08:59.482878 25336 consensus_queue.cc:237] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.482759 24899 catalog_manager.cc:5495] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 reported cstate change: term changed from 0 to 1, leader changed from <none> to 720dbff207a94212bdaf227d9b05d627 (127.23.120.129). New cstate: current_term: 1 leader_uuid: "720dbff207a94212bdaf227d9b05d627" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } health_report { overall_health: HEALTHY } } }
I20260807 11:08:59.479154 25368 raft_consensus.cc:504] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.487488 25048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59d5a742d82444391f7103f701851bf" candidate_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720dbff207a94212bdaf227d9b05d627" is_pre_election: true
W20260807 11:08:59.488399 24993 leader_election.cc:341] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:59.488662 25323 tablet_bootstrap.cc:492] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf: Bootstrap starting.
I20260807 11:08:59.489091 24993 leader_election.cc:302] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 720dbff207a94212bdaf227d9b05d627; no voters: 3751e3c65b0340eeba13019e16fcc3bf, bbeb19c5189a46bcb4c327db2b6bc861
W20260807 11:08:59.489410 25096 leader_election.cc:341] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:59.490688 25334 raft_consensus.cc:2726] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:08:59.491639 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59d5a742d82444391f7103f701851bf" candidate_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf" is_pre_election: true
I20260807 11:08:59.493024 25333 raft_consensus.cc:482] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:08:59.493386 25095 leader_election.cc:341] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:59.493517 25333 raft_consensus.cc:504] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.493929 25095 leader_election.cc:302] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbeb19c5189a46bcb4c327db2b6bc861; no voters: 3751e3c65b0340eeba13019e16fcc3bf, 720dbff207a94212bdaf227d9b05d627
I20260807 11:08:59.494158 25325 tablet_bootstrap.cc:492] T dfd144f74da942029b3904cc9cfda2a2 P 720dbff207a94212bdaf227d9b05d627: No bootstrap required, opened a new log
I20260807 11:08:59.494915 25325 ts_tablet_manager.cc:1397] T dfd144f74da942029b3904cc9cfda2a2 P 720dbff207a94212bdaf227d9b05d627: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.014s
I20260807 11:08:59.495788 25323 tablet_bootstrap.cc:654] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.495678 25368 leader_election.cc:288] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:08:59.497515 25368 raft_consensus.cc:2726] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:08:59.498306 25325 raft_consensus.cc:348] T dfd144f74da942029b3904cc9cfda2a2 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.499222 25325 raft_consensus.cc:374] T dfd144f74da942029b3904cc9cfda2a2 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.499647 25325 raft_consensus.cc:729] T dfd144f74da942029b3904cc9cfda2a2 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720dbff207a94212bdaf227d9b05d627, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.500502 25325 consensus_queue.cc:260] T dfd144f74da942029b3904cc9cfda2a2 P 720dbff207a94212bdaf227d9b05d627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.503911 25325 ts_tablet_manager.cc:1428] T dfd144f74da942029b3904cc9cfda2a2 P 720dbff207a94212bdaf227d9b05d627: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260807 11:08:59.505227 25325 tablet_bootstrap.cc:492] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627: Bootstrap starting.
I20260807 11:08:59.505649 25333 leader_election.cc:288] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989), bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315)
I20260807 11:08:59.506150 25324 tablet_bootstrap.cc:654] T fd44192986994fc086a65a73dea598d6 P bbeb19c5189a46bcb4c327db2b6bc861: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.513307 25325 tablet_bootstrap.cc:654] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.521394 25324 tablet_bootstrap.cc:492] T fd44192986994fc086a65a73dea598d6 P bbeb19c5189a46bcb4c327db2b6bc861: No bootstrap required, opened a new log
I20260807 11:08:59.522033 25324 ts_tablet_manager.cc:1397] T fd44192986994fc086a65a73dea598d6 P bbeb19c5189a46bcb4c327db2b6bc861: Time spent bootstrapping tablet: real 0.039s user 0.013s sys 0.003s
I20260807 11:08:59.525835 25324 raft_consensus.cc:348] T fd44192986994fc086a65a73dea598d6 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.537688 25324 raft_consensus.cc:374] T fd44192986994fc086a65a73dea598d6 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.538069 25324 raft_consensus.cc:729] T fd44192986994fc086a65a73dea598d6 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbeb19c5189a46bcb4c327db2b6bc861, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.540006 25324 consensus_queue.cc:260] T fd44192986994fc086a65a73dea598d6 P bbeb19c5189a46bcb4c327db2b6bc861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.545480 25324 ts_tablet_manager.cc:1428] T fd44192986994fc086a65a73dea598d6 P bbeb19c5189a46bcb4c327db2b6bc861: Time spent starting tablet: real 0.023s user 0.009s sys 0.000s
I20260807 11:08:59.546578 25324 tablet_bootstrap.cc:492] T 1924e8081c31446c8953300614a93d43 P bbeb19c5189a46bcb4c327db2b6bc861: Bootstrap starting.
I20260807 11:08:59.549162 24900 catalog_manager.cc:5495] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130). New cstate: current_term: 1 leader_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } health_report { overall_health: UNKNOWN } } }
I20260807 11:08:59.550418 25334 raft_consensus.cc:482] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:59.551012 25334 raft_consensus.cc:504] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.555002 25334 leader_election.cc:288] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:08:59.556151 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425296c8f30a46e0a77af3752db337bf" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf" is_pre_election: true
I20260807 11:08:59.557654 25261 raft_consensus.cc:2454] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 720dbff207a94212bdaf227d9b05d627 in term 0.
I20260807 11:08:59.557778 25153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425296c8f30a46e0a77af3752db337bf" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" is_pre_election: true
I20260807 11:08:59.558789 24993 leader_election.cc:302] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, 720dbff207a94212bdaf227d9b05d627; no voters:
W20260807 11:08:59.559427 24990 leader_election.cc:341] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:59.559741 25334 raft_consensus.cc:2781] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:08:59.560143 25334 raft_consensus.cc:482] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:08:59.560572 25334 raft_consensus.cc:3037] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:59.557197 25324 tablet_bootstrap.cc:654] T 1924e8081c31446c8953300614a93d43 P bbeb19c5189a46bcb4c327db2b6bc861: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.569252 25048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfd144f74da942029b3904cc9cfda2a2" candidate_uuid: "3751e3c65b0340eeba13019e16fcc3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720dbff207a94212bdaf227d9b05d627" is_pre_election: true
I20260807 11:08:59.569944 25048 raft_consensus.cc:2454] T dfd144f74da942029b3904cc9cfda2a2 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3751e3c65b0340eeba13019e16fcc3bf in term 0.
I20260807 11:08:59.570411 25325 tablet_bootstrap.cc:492] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627: No bootstrap required, opened a new log
I20260807 11:08:59.570930 25325 ts_tablet_manager.cc:1397] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627: Time spent bootstrapping tablet: real 0.066s user 0.013s sys 0.041s
I20260807 11:08:59.571306 25210 leader_election.cc:302] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, 720dbff207a94212bdaf227d9b05d627; no voters:
I20260807 11:08:59.572434 25333 raft_consensus.cc:2781] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:08:59.572851 25333 raft_consensus.cc:482] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:08:59.572566 25334 raft_consensus.cc:504] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.574321 25153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfd144f74da942029b3904cc9cfda2a2" candidate_uuid: "3751e3c65b0340eeba13019e16fcc3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" is_pre_election: true
I20260807 11:08:59.574061 25325 raft_consensus.cc:348] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.574919 25325 raft_consensus.cc:374] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.574957 25153 raft_consensus.cc:2454] T dfd144f74da942029b3904cc9cfda2a2 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3751e3c65b0340eeba13019e16fcc3bf in term 0.
I20260807 11:08:59.575407 25325 raft_consensus.cc:729] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720dbff207a94212bdaf227d9b05d627, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.573244 25333 raft_consensus.cc:3037] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:59.576298 25325 consensus_queue.cc:260] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.579584 25325 ts_tablet_manager.cc:1428] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260807 11:08:59.580591 25325 tablet_bootstrap.cc:492] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627: Bootstrap starting.
I20260807 11:08:59.581177 25153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425296c8f30a46e0a77af3752db337bf" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
I20260807 11:08:59.577051 25323 tablet_bootstrap.cc:492] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf: No bootstrap required, opened a new log
I20260807 11:08:59.581907 25323 ts_tablet_manager.cc:1397] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf: Time spent bootstrapping tablet: real 0.113s user 0.019s sys 0.020s
I20260807 11:08:59.582017 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425296c8f30a46e0a77af3752db337bf" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
W20260807 11:08:59.582471 24990 leader_election.cc:341] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:59.582647 25261 raft_consensus.cc:3037] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:59.584547 25323 raft_consensus.cc:348] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.585299 25323 raft_consensus.cc:374] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.584955 25334 leader_election.cc:288] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 1 election: Requested vote from peers bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:08:59.586143 25333 raft_consensus.cc:504] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.589412 25153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfd144f74da942029b3904cc9cfda2a2" candidate_uuid: "3751e3c65b0340eeba13019e16fcc3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
I20260807 11:08:59.589555 25261 raft_consensus.cc:2454] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720dbff207a94212bdaf227d9b05d627 in term 1.
I20260807 11:08:59.590009 25153 raft_consensus.cc:3037] T dfd144f74da942029b3904cc9cfda2a2 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:59.591005 25325 tablet_bootstrap.cc:654] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.590737 24993 leader_election.cc:302] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, 720dbff207a94212bdaf227d9b05d627; no voters: bbeb19c5189a46bcb4c327db2b6bc861
I20260807 11:08:59.592089 25334 raft_consensus.cc:2781] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:08:59.592846 25334 raft_consensus.cc:686] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [term 1 LEADER]: Becoming Leader. State: Replica: 720dbff207a94212bdaf227d9b05d627, State: Running, Role: LEADER
I20260807 11:08:59.585597 25323 raft_consensus.cc:729] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3751e3c65b0340eeba13019e16fcc3bf, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.593883 25334 consensus_queue.cc:237] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.594542 25323 consensus_queue.cc:260] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.598629 25048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfd144f74da942029b3904cc9cfda2a2" candidate_uuid: "3751e3c65b0340eeba13019e16fcc3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720dbff207a94212bdaf227d9b05d627"
I20260807 11:08:59.599350 25048 raft_consensus.cc:3037] T dfd144f74da942029b3904cc9cfda2a2 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:59.600935 25324 tablet_bootstrap.cc:492] T 1924e8081c31446c8953300614a93d43 P bbeb19c5189a46bcb4c327db2b6bc861: No bootstrap required, opened a new log
I20260807 11:08:59.601457 25324 ts_tablet_manager.cc:1397] T 1924e8081c31446c8953300614a93d43 P bbeb19c5189a46bcb4c327db2b6bc861: Time spent bootstrapping tablet: real 0.055s user 0.016s sys 0.000s
I20260807 11:08:59.604635 25324 raft_consensus.cc:348] T 1924e8081c31446c8953300614a93d43 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.606336 25324 raft_consensus.cc:374] T 1924e8081c31446c8953300614a93d43 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.606681 25324 raft_consensus.cc:729] T 1924e8081c31446c8953300614a93d43 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbeb19c5189a46bcb4c327db2b6bc861, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.607491 25324 consensus_queue.cc:260] T 1924e8081c31446c8953300614a93d43 P bbeb19c5189a46bcb4c327db2b6bc861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.610222 25324 ts_tablet_manager.cc:1428] T 1924e8081c31446c8953300614a93d43 P bbeb19c5189a46bcb4c327db2b6bc861: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260807 11:08:59.611166 25324 tablet_bootstrap.cc:492] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861: Bootstrap starting.
I20260807 11:08:59.614555 25048 raft_consensus.cc:2454] T dfd144f74da942029b3904cc9cfda2a2 P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3751e3c65b0340eeba13019e16fcc3bf in term 1.
I20260807 11:08:59.615919 25210 leader_election.cc:302] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, 720dbff207a94212bdaf227d9b05d627; no voters:
I20260807 11:08:59.617461 25324 tablet_bootstrap.cc:654] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.620865 25333 leader_election.cc:288] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 1 election: Requested vote from peers 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989), bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315)
I20260807 11:08:59.625025 25333 raft_consensus.cc:2781] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:08:59.625571 25333 raft_consensus.cc:686] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 LEADER]: Becoming Leader. State: Replica: 3751e3c65b0340eeba13019e16fcc3bf, State: Running, Role: LEADER
I20260807 11:08:59.626571 25333 consensus_queue.cc:237] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.628715 25336 raft_consensus.cc:482] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:59.629225 25336 raft_consensus.cc:504] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.631318 25336 leader_election.cc:288] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:08:59.630991 24897 catalog_manager.cc:5495] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 reported cstate change: term changed from 0 to 1, leader changed from <none> to 720dbff207a94212bdaf227d9b05d627 (127.23.120.129). New cstate: current_term: 1 leader_uuid: "720dbff207a94212bdaf227d9b05d627" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } health_report { overall_health: UNKNOWN } } }
I20260807 11:08:59.633723 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c732ca6d3a44758b2f1f46d56f9161" candidate_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf" is_pre_election: true
I20260807 11:08:59.634107 25047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c732ca6d3a44758b2f1f46d56f9161" candidate_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720dbff207a94212bdaf227d9b05d627" is_pre_election: true
I20260807 11:08:59.634569 25261 raft_consensus.cc:2454] T 39c732ca6d3a44758b2f1f46d56f9161 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbeb19c5189a46bcb4c327db2b6bc861 in term 0.
I20260807 11:08:59.634948 25047 raft_consensus.cc:2454] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbeb19c5189a46bcb4c327db2b6bc861 in term 0.
I20260807 11:08:59.635943 25095 leader_election.cc:302] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, bbeb19c5189a46bcb4c327db2b6bc861; no voters:
I20260807 11:08:59.637635 25153 raft_consensus.cc:2454] T dfd144f74da942029b3904cc9cfda2a2 P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3751e3c65b0340eeba13019e16fcc3bf in term 1.
I20260807 11:08:59.640609 25336 raft_consensus.cc:2781] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:08:59.641023 25336 raft_consensus.cc:482] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:08:59.641374 25336 raft_consensus.cc:3037] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:59.648586 25336 raft_consensus.cc:504] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.650928 25323 ts_tablet_manager.cc:1428] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf: Time spent starting tablet: real 0.069s user 0.021s sys 0.012s
I20260807 11:08:59.651239 25048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c732ca6d3a44758b2f1f46d56f9161" candidate_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720dbff207a94212bdaf227d9b05d627"
I20260807 11:08:59.651458 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c732ca6d3a44758b2f1f46d56f9161" candidate_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
I20260807 11:08:59.651933 25048 raft_consensus.cc:3037] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:59.652025 25261 raft_consensus.cc:3037] T 39c732ca6d3a44758b2f1f46d56f9161 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:59.652043 25323 tablet_bootstrap.cc:492] T fd44192986994fc086a65a73dea598d6 P 3751e3c65b0340eeba13019e16fcc3bf: Bootstrap starting.
I20260807 11:08:59.651222 25336 leader_election.cc:288] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 1 election: Requested vote from peers 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:08:59.658993 25323 tablet_bootstrap.cc:654] T fd44192986994fc086a65a73dea598d6 P 3751e3c65b0340eeba13019e16fcc3bf: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.659749 25261 raft_consensus.cc:2454] T 39c732ca6d3a44758b2f1f46d56f9161 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbeb19c5189a46bcb4c327db2b6bc861 in term 1.
I20260807 11:08:59.658777 24899 catalog_manager.cc:5495] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131). New cstate: current_term: 1 leader_uuid: "3751e3c65b0340eeba13019e16fcc3bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } health_report { overall_health: HEALTHY } } }
I20260807 11:08:59.661029 25324 tablet_bootstrap.cc:492] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861: No bootstrap required, opened a new log
I20260807 11:08:59.661183 25095 leader_election.cc:302] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, bbeb19c5189a46bcb4c327db2b6bc861; no voters:
I20260807 11:08:59.661691 25324 ts_tablet_manager.cc:1397] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861: Time spent bootstrapping tablet: real 0.051s user 0.012s sys 0.009s
I20260807 11:08:59.662277 25325 tablet_bootstrap.cc:492] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627: No bootstrap required, opened a new log
I20260807 11:08:59.662742 25325 ts_tablet_manager.cc:1397] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627: Time spent bootstrapping tablet: real 0.082s user 0.015s sys 0.020s
I20260807 11:08:59.664309 25324 raft_consensus.cc:348] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.664881 25325 raft_consensus.cc:348] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.665477 25325 raft_consensus.cc:374] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.665674 25325 raft_consensus.cc:729] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720dbff207a94212bdaf227d9b05d627, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.666139 25325 consensus_queue.cc:260] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.666515 25336 raft_consensus.cc:2781] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:08:59.666981 25336 raft_consensus.cc:686] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 LEADER]: Becoming Leader. State: Replica: bbeb19c5189a46bcb4c327db2b6bc861, State: Running, Role: LEADER
I20260807 11:08:59.668210 25325 ts_tablet_manager.cc:1428] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260807 11:08:59.667769 25336 consensus_queue.cc:237] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.670305 25325 tablet_bootstrap.cc:492] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627: Bootstrap starting.
I20260807 11:08:59.669528 25048 raft_consensus.cc:2454] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbeb19c5189a46bcb4c327db2b6bc861 in term 1.
I20260807 11:08:59.672693 25324 raft_consensus.cc:374] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.673010 25324 raft_consensus.cc:729] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbeb19c5189a46bcb4c327db2b6bc861, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.673626 25324 consensus_queue.cc:260] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.676581 25324 ts_tablet_manager.cc:1428] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260807 11:08:59.676730 25325 tablet_bootstrap.cc:654] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.677608 25324 tablet_bootstrap.cc:492] T 425296c8f30a46e0a77af3752db337bf P bbeb19c5189a46bcb4c327db2b6bc861: Bootstrap starting.
I20260807 11:08:59.682770 25324 tablet_bootstrap.cc:654] T 425296c8f30a46e0a77af3752db337bf P bbeb19c5189a46bcb4c327db2b6bc861: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.684091 25334 raft_consensus.cc:482] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:59.689417 25334 raft_consensus.cc:504] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.691942 25334 leader_election.cc:288] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:08:59.693058 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd44192986994fc086a65a73dea598d6" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf" is_pre_election: true
I20260807 11:08:59.693377 25153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd44192986994fc086a65a73dea598d6" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" is_pre_election: true
I20260807 11:08:59.693938 25153 raft_consensus.cc:2454] T fd44192986994fc086a65a73dea598d6 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 720dbff207a94212bdaf227d9b05d627 in term 0.
W20260807 11:08:59.694548 24993 leader_election.cc:341] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:59.695047 24990 leader_election.cc:302] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 720dbff207a94212bdaf227d9b05d627, bbeb19c5189a46bcb4c327db2b6bc861; no voters: 3751e3c65b0340eeba13019e16fcc3bf
I20260807 11:08:59.696496 25334 raft_consensus.cc:2781] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:08:59.696947 25334 raft_consensus.cc:482] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:08:59.697372 25334 raft_consensus.cc:3037] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:59.702106 25333 raft_consensus.cc:482] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:59.702556 25333 raft_consensus.cc:504] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.704564 25333 leader_election.cc:288] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989), bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315)
I20260807 11:08:59.708225 25048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89018784373e4903ae1a1bcaa1ac3f4e" candidate_uuid: "3751e3c65b0340eeba13019e16fcc3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720dbff207a94212bdaf227d9b05d627" is_pre_election: true
I20260807 11:08:59.706875 25153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89018784373e4903ae1a1bcaa1ac3f4e" candidate_uuid: "3751e3c65b0340eeba13019e16fcc3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" is_pre_election: true
I20260807 11:08:59.709316 25048 raft_consensus.cc:2454] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3751e3c65b0340eeba13019e16fcc3bf in term 0.
W20260807 11:08:59.710654 25207 leader_election.cc:341] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:59.711406 25210 leader_election.cc:302] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, 720dbff207a94212bdaf227d9b05d627; no voters: bbeb19c5189a46bcb4c327db2b6bc861
I20260807 11:08:59.713028 25333 raft_consensus.cc:2781] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:08:59.713469 25333 raft_consensus.cc:482] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:08:59.713872 25333 raft_consensus.cc:3037] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:59.708088 25334 raft_consensus.cc:504] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.720353 25323 tablet_bootstrap.cc:492] T fd44192986994fc086a65a73dea598d6 P 3751e3c65b0340eeba13019e16fcc3bf: No bootstrap required, opened a new log
I20260807 11:08:59.719844 24899 catalog_manager.cc:5495] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130). New cstate: current_term: 1 leader_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } health_report { overall_health: UNKNOWN } } }
I20260807 11:08:59.721010 25323 ts_tablet_manager.cc:1397] T fd44192986994fc086a65a73dea598d6 P 3751e3c65b0340eeba13019e16fcc3bf: Time spent bootstrapping tablet: real 0.069s user 0.020s sys 0.010s
I20260807 11:08:59.725713 25325 tablet_bootstrap.cc:492] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627: No bootstrap required, opened a new log
I20260807 11:08:59.725911 25153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd44192986994fc086a65a73dea598d6" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
I20260807 11:08:59.726194 25325 ts_tablet_manager.cc:1397] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627: Time spent bootstrapping tablet: real 0.056s user 0.014s sys 0.025s
I20260807 11:08:59.726470 25153 raft_consensus.cc:3037] T fd44192986994fc086a65a73dea598d6 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:59.726810 25323 raft_consensus.cc:348] T fd44192986994fc086a65a73dea598d6 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.727639 25323 raft_consensus.cc:374] T fd44192986994fc086a65a73dea598d6 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.727993 25323 raft_consensus.cc:729] T fd44192986994fc086a65a73dea598d6 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3751e3c65b0340eeba13019e16fcc3bf, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.728879 25323 consensus_queue.cc:260] T fd44192986994fc086a65a73dea598d6 P 3751e3c65b0340eeba13019e16fcc3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.729017 25325 raft_consensus.cc:348] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.729846 25325 raft_consensus.cc:374] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.733659 25325 raft_consensus.cc:729] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720dbff207a94212bdaf227d9b05d627, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.734449 25333 raft_consensus.cc:504] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.738049 25324 tablet_bootstrap.cc:492] T 425296c8f30a46e0a77af3752db337bf P bbeb19c5189a46bcb4c327db2b6bc861: No bootstrap required, opened a new log
I20260807 11:08:59.738541 25324 ts_tablet_manager.cc:1397] T 425296c8f30a46e0a77af3752db337bf P bbeb19c5189a46bcb4c327db2b6bc861: Time spent bootstrapping tablet: real 0.061s user 0.010s sys 0.015s
I20260807 11:08:59.738798 25048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89018784373e4903ae1a1bcaa1ac3f4e" candidate_uuid: "3751e3c65b0340eeba13019e16fcc3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720dbff207a94212bdaf227d9b05d627"
I20260807 11:08:59.739404 25048 raft_consensus.cc:3037] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:59.741192 25324 raft_consensus.cc:348] T 425296c8f30a46e0a77af3752db337bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.734360 25325 consensus_queue.cc:260] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.731465 25323 ts_tablet_manager.cc:1428] T fd44192986994fc086a65a73dea598d6 P 3751e3c65b0340eeba13019e16fcc3bf: Time spent starting tablet: real 0.010s user 0.002s sys 0.004s
I20260807 11:08:59.742457 25152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89018784373e4903ae1a1bcaa1ac3f4e" candidate_uuid: "3751e3c65b0340eeba13019e16fcc3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
I20260807 11:08:59.743711 25323 tablet_bootstrap.cc:492] T c1eee668b4cc4eddad42edaf1827309d P 3751e3c65b0340eeba13019e16fcc3bf: Bootstrap starting.
I20260807 11:08:59.745982 25048 raft_consensus.cc:2454] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3751e3c65b0340eeba13019e16fcc3bf in term 1.
I20260807 11:08:59.744002 25334 leader_election.cc:288] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 1 election: Requested vote from peers bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
W20260807 11:08:59.746222 25207 leader_election.cc:341] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:08:59.747087 25210 leader_election.cc:302] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, 720dbff207a94212bdaf227d9b05d627; no voters: bbeb19c5189a46bcb4c327db2b6bc861
I20260807 11:08:59.748149 25386 raft_consensus.cc:2781] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:08:59.741267 25333 leader_election.cc:288] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 1 election: Requested vote from peers 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989), bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315)
I20260807 11:08:59.748641 25386 raft_consensus.cc:686] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [term 1 LEADER]: Becoming Leader. State: Replica: 3751e3c65b0340eeba13019e16fcc3bf, State: Running, Role: LEADER
I20260807 11:08:59.744926 25324 raft_consensus.cc:374] T 425296c8f30a46e0a77af3752db337bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.745132 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd44192986994fc086a65a73dea598d6" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
I20260807 11:08:59.749555 25261 raft_consensus.cc:3037] T fd44192986994fc086a65a73dea598d6 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:59.749579 25386 consensus_queue.cc:237] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.735118 25153 raft_consensus.cc:2454] T fd44192986994fc086a65a73dea598d6 P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720dbff207a94212bdaf227d9b05d627 in term 1.
I20260807 11:08:59.753680 25323 tablet_bootstrap.cc:654] T c1eee668b4cc4eddad42edaf1827309d P 3751e3c65b0340eeba13019e16fcc3bf: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.754285 24990 leader_election.cc:302] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 720dbff207a94212bdaf227d9b05d627, bbeb19c5189a46bcb4c327db2b6bc861; no voters:
I20260807 11:08:59.755203 25334 raft_consensus.cc:2781] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:08:59.755688 25334 raft_consensus.cc:686] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [term 1 LEADER]: Becoming Leader. State: Replica: 720dbff207a94212bdaf227d9b05d627, State: Running, Role: LEADER
I20260807 11:08:59.745663 25325 ts_tablet_manager.cc:1428] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260807 11:08:59.756533 25334 consensus_queue.cc:237] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.757568 25325 tablet_bootstrap.cc:492] T 5e25a8abd35e45c2af9bdf5f6b053269 P 720dbff207a94212bdaf227d9b05d627: Bootstrap starting.
I20260807 11:08:59.758882 25261 raft_consensus.cc:2454] T fd44192986994fc086a65a73dea598d6 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720dbff207a94212bdaf227d9b05d627 in term 1.
I20260807 11:08:59.759507 25324 raft_consensus.cc:729] T 425296c8f30a46e0a77af3752db337bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbeb19c5189a46bcb4c327db2b6bc861, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.760303 25324 consensus_queue.cc:260] T 425296c8f30a46e0a77af3752db337bf P bbeb19c5189a46bcb4c327db2b6bc861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.768404 25325 tablet_bootstrap.cc:654] T 5e25a8abd35e45c2af9bdf5f6b053269 P 720dbff207a94212bdaf227d9b05d627: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.768978 25323 tablet_bootstrap.cc:492] T c1eee668b4cc4eddad42edaf1827309d P 3751e3c65b0340eeba13019e16fcc3bf: No bootstrap required, opened a new log
I20260807 11:08:59.769492 25324 ts_tablet_manager.cc:1428] T 425296c8f30a46e0a77af3752db337bf P bbeb19c5189a46bcb4c327db2b6bc861: Time spent starting tablet: real 0.031s user 0.006s sys 0.003s
I20260807 11:08:59.769543 25323 ts_tablet_manager.cc:1397] T c1eee668b4cc4eddad42edaf1827309d P 3751e3c65b0340eeba13019e16fcc3bf: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.006s
I20260807 11:08:59.770591 25324 tablet_bootstrap.cc:492] T 89018784373e4903ae1a1bcaa1ac3f4e P bbeb19c5189a46bcb4c327db2b6bc861: Bootstrap starting.
I20260807 11:08:59.772591 25323 raft_consensus.cc:348] T c1eee668b4cc4eddad42edaf1827309d P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.773375 25323 raft_consensus.cc:374] T c1eee668b4cc4eddad42edaf1827309d P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.773643 25323 raft_consensus.cc:729] T c1eee668b4cc4eddad42edaf1827309d P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3751e3c65b0340eeba13019e16fcc3bf, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.774318 25323 consensus_queue.cc:260] T c1eee668b4cc4eddad42edaf1827309d P 3751e3c65b0340eeba13019e16fcc3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.776736 25323 ts_tablet_manager.cc:1428] T c1eee668b4cc4eddad42edaf1827309d P 3751e3c65b0340eeba13019e16fcc3bf: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260807 11:08:59.777678 25323 tablet_bootstrap.cc:492] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf: Bootstrap starting.
I20260807 11:08:59.776708 24898 catalog_manager.cc:5495] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131). New cstate: current_term: 1 leader_uuid: "3751e3c65b0340eeba13019e16fcc3bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } health_report { overall_health: HEALTHY } } }
I20260807 11:08:59.781008 25324 tablet_bootstrap.cc:654] T 89018784373e4903ae1a1bcaa1ac3f4e P bbeb19c5189a46bcb4c327db2b6bc861: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.782931 25323 tablet_bootstrap.cc:654] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.794690 25324 tablet_bootstrap.cc:492] T 89018784373e4903ae1a1bcaa1ac3f4e P bbeb19c5189a46bcb4c327db2b6bc861: No bootstrap required, opened a new log
I20260807 11:08:59.795362 25324 ts_tablet_manager.cc:1397] T 89018784373e4903ae1a1bcaa1ac3f4e P bbeb19c5189a46bcb4c327db2b6bc861: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.009s
I20260807 11:08:59.794761 24899 catalog_manager.cc:5495] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 reported cstate change: term changed from 0 to 1, leader changed from <none> to 720dbff207a94212bdaf227d9b05d627 (127.23.120.129). New cstate: current_term: 1 leader_uuid: "720dbff207a94212bdaf227d9b05d627" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } health_report { overall_health: UNKNOWN } } }
I20260807 11:08:59.798452 25324 raft_consensus.cc:348] T 89018784373e4903ae1a1bcaa1ac3f4e P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.799154 25324 raft_consensus.cc:374] T 89018784373e4903ae1a1bcaa1ac3f4e P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.799437 25324 raft_consensus.cc:729] T 89018784373e4903ae1a1bcaa1ac3f4e P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbeb19c5189a46bcb4c327db2b6bc861, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.800055 25324 consensus_queue.cc:260] T 89018784373e4903ae1a1bcaa1ac3f4e P bbeb19c5189a46bcb4c327db2b6bc861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.798872 25323 tablet_bootstrap.cc:492] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf: No bootstrap required, opened a new log
I20260807 11:08:59.801033 25323 ts_tablet_manager.cc:1397] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.013s
I20260807 11:08:59.803133 25323 raft_consensus.cc:348] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.803786 25323 raft_consensus.cc:374] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.804035 25323 raft_consensus.cc:729] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3751e3c65b0340eeba13019e16fcc3bf, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.804373 25325 tablet_bootstrap.cc:492] T 5e25a8abd35e45c2af9bdf5f6b053269 P 720dbff207a94212bdaf227d9b05d627: No bootstrap required, opened a new log
I20260807 11:08:59.804823 25325 ts_tablet_manager.cc:1397] T 5e25a8abd35e45c2af9bdf5f6b053269 P 720dbff207a94212bdaf227d9b05d627: Time spent bootstrapping tablet: real 0.047s user 0.012s sys 0.011s
I20260807 11:08:59.804642 25323 consensus_queue.cc:260] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.806664 25323 ts_tablet_manager.cc:1428] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260807 11:08:59.807114 25325 raft_consensus.cc:348] T 5e25a8abd35e45c2af9bdf5f6b053269 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.807762 25325 raft_consensus.cc:374] T 5e25a8abd35e45c2af9bdf5f6b053269 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.808053 25325 raft_consensus.cc:729] T 5e25a8abd35e45c2af9bdf5f6b053269 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720dbff207a94212bdaf227d9b05d627, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.807511 25323 tablet_bootstrap.cc:492] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf: Bootstrap starting.
I20260807 11:08:59.808616 25325 consensus_queue.cc:260] T 5e25a8abd35e45c2af9bdf5f6b053269 P 720dbff207a94212bdaf227d9b05d627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.810523 25325 ts_tablet_manager.cc:1428] T 5e25a8abd35e45c2af9bdf5f6b053269 P 720dbff207a94212bdaf227d9b05d627: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20260807 11:08:59.811501 25325 tablet_bootstrap.cc:492] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627: Bootstrap starting.
I20260807 11:08:59.811539 25324 ts_tablet_manager.cc:1428] T 89018784373e4903ae1a1bcaa1ac3f4e P bbeb19c5189a46bcb4c327db2b6bc861: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20260807 11:08:59.816823 25323 tablet_bootstrap.cc:654] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.817001 25325 tablet_bootstrap.cc:654] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627: Neither blocks nor log segments found. Creating new log.
I20260807 11:08:59.829105 25325 tablet_bootstrap.cc:492] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627: No bootstrap required, opened a new log
I20260807 11:08:59.829540 25325 ts_tablet_manager.cc:1397] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.007s
I20260807 11:08:59.830840 25323 tablet_bootstrap.cc:492] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf: No bootstrap required, opened a new log
I20260807 11:08:59.831250 25323 ts_tablet_manager.cc:1397] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.011s
I20260807 11:08:59.831890 25325 raft_consensus.cc:348] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.832561 25325 raft_consensus.cc:374] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.832841 25325 raft_consensus.cc:729] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720dbff207a94212bdaf227d9b05d627, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.833436 25325 consensus_queue.cc:260] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.833596 25323 raft_consensus.cc:348] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.834205 25323 raft_consensus.cc:374] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:08:59.834477 25323 raft_consensus.cc:729] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3751e3c65b0340eeba13019e16fcc3bf, State: Initialized, Role: FOLLOWER
I20260807 11:08:59.835637 25323 consensus_queue.cc:260] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.836072 25325 ts_tablet_manager.cc:1428] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260807 11:08:59.837698 25323 ts_tablet_manager.cc:1428] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260807 11:08:59.867920 25386 raft_consensus.cc:482] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:59.868400 25386 raft_consensus.cc:504] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.870208 25386 leader_election.cc:288] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989), bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315)
I20260807 11:08:59.871048 25048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59d5a742d82444391f7103f701851bf" candidate_uuid: "3751e3c65b0340eeba13019e16fcc3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720dbff207a94212bdaf227d9b05d627" is_pre_election: true
I20260807 11:08:59.871222 25153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59d5a742d82444391f7103f701851bf" candidate_uuid: "3751e3c65b0340eeba13019e16fcc3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" is_pre_election: true
I20260807 11:08:59.871583 25048 raft_consensus.cc:2454] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3751e3c65b0340eeba13019e16fcc3bf in term 0.
I20260807 11:08:59.871795 25153 raft_consensus.cc:2454] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3751e3c65b0340eeba13019e16fcc3bf in term 0.
I20260807 11:08:59.872726 25210 leader_election.cc:302] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, 720dbff207a94212bdaf227d9b05d627; no voters:
I20260807 11:08:59.873481 25386 raft_consensus.cc:2781] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:08:59.873799 25386 raft_consensus.cc:482] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:08:59.874074 25386 raft_consensus.cc:3037] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:59.879000 25386 raft_consensus.cc:504] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.880560 25386 leader_election.cc:288] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 1 election: Requested vote from peers 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989), bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315)
I20260807 11:08:59.881553 25048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59d5a742d82444391f7103f701851bf" candidate_uuid: "3751e3c65b0340eeba13019e16fcc3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720dbff207a94212bdaf227d9b05d627"
I20260807 11:08:59.881655 25153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59d5a742d82444391f7103f701851bf" candidate_uuid: "3751e3c65b0340eeba13019e16fcc3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
I20260807 11:08:59.882148 25048 raft_consensus.cc:3037] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:59.882226 25153 raft_consensus.cc:3037] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:59.886754 25153 raft_consensus.cc:2454] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3751e3c65b0340eeba13019e16fcc3bf in term 1.
I20260807 11:08:59.887107 25048 raft_consensus.cc:2454] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3751e3c65b0340eeba13019e16fcc3bf in term 1.
I20260807 11:08:59.887669 25207 leader_election.cc:302] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, bbeb19c5189a46bcb4c327db2b6bc861; no voters:
I20260807 11:08:59.888523 25386 raft_consensus.cc:2781] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:08:59.889063 25386 raft_consensus.cc:686] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [term 1 LEADER]: Becoming Leader. State: Replica: 3751e3c65b0340eeba13019e16fcc3bf, State: Running, Role: LEADER
I20260807 11:08:59.889916 25386 consensus_queue.cc:237] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.893324 25334 consensus_queue.cc:1035] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:08:59.899217 24899 catalog_manager.cc:5495] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131). New cstate: current_term: 1 leader_uuid: "3751e3c65b0340eeba13019e16fcc3bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } health_report { overall_health: HEALTHY } } }
I20260807 11:08:59.907111 25362 consensus_queue.cc:1035] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:08:59.912114 25336 consensus_queue.cc:1035] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:08:59.930490 25048 raft_consensus.cc:3037] T 5e25a8abd35e45c2af9bdf5f6b053269 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:59.933607 25370 raft_consensus.cc:482] T 425296c8f30a46e0a77af3752db337bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:08:59.934139 25370 raft_consensus.cc:504] T 425296c8f30a46e0a77af3752db337bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:08:59.937178 25370 leader_election.cc:288] T 425296c8f30a46e0a77af3752db337bf P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:08:59.937343 25047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425296c8f30a46e0a77af3752db337bf" candidate_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720dbff207a94212bdaf227d9b05d627" is_pre_election: true
I20260807 11:08:59.937379 25368 consensus_queue.cc:1035] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:08:59.938169 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425296c8f30a46e0a77af3752db337bf" candidate_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf" is_pre_election: true
I20260807 11:08:59.939029 25260 raft_consensus.cc:2379] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbeb19c5189a46bcb4c327db2b6bc861 in current term 1: Already voted for candidate 720dbff207a94212bdaf227d9b05d627 in this term.
I20260807 11:08:59.940106 25095 leader_election.cc:302] T 425296c8f30a46e0a77af3752db337bf P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbeb19c5189a46bcb4c327db2b6bc861; no voters: 3751e3c65b0340eeba13019e16fcc3bf, 720dbff207a94212bdaf227d9b05d627
I20260807 11:08:59.941365 25368 raft_consensus.cc:3037] T 425296c8f30a46e0a77af3752db337bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:08:59.948935 25368 raft_consensus.cc:2726] T 425296c8f30a46e0a77af3752db337bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:08:59.983860 25362 consensus_queue.cc:1035] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:08:59.995435 25334 consensus_queue.cc:1035] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:09:00.029400 25368 raft_consensus.cc:482] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:00.029870 25368 raft_consensus.cc:504] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:00.031559 25368 leader_election.cc:288] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:09:00.032485 25047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1eee668b4cc4eddad42edaf1827309d" candidate_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720dbff207a94212bdaf227d9b05d627" is_pre_election: true
I20260807 11:09:00.032752 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1eee668b4cc4eddad42edaf1827309d" candidate_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf" is_pre_election: true
I20260807 11:09:00.033088 25390 raft_consensus.cc:482] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:00.033258 25047 raft_consensus.cc:2454] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbeb19c5189a46bcb4c327db2b6bc861 in term 0.
I20260807 11:09:00.033469 25260 raft_consensus.cc:2454] T c1eee668b4cc4eddad42edaf1827309d P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbeb19c5189a46bcb4c327db2b6bc861 in term 0.
I20260807 11:09:00.033774 25334 raft_consensus.cc:482] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:00.033608 25390 raft_consensus.cc:504] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:00.034217 25334 raft_consensus.cc:504] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:00.034606 25096 leader_election.cc:302] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 720dbff207a94212bdaf227d9b05d627, bbeb19c5189a46bcb4c327db2b6bc861; no voters:
I20260807 11:09:00.035528 25368 raft_consensus.cc:2781] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:00.035882 25368 raft_consensus.cc:482] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:00.035924 25390 leader_election.cc:288] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989), bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315)
I20260807 11:09:00.036355 25368 raft_consensus.cc:3037] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:00.036957 25047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1924e8081c31446c8953300614a93d43" candidate_uuid: "3751e3c65b0340eeba13019e16fcc3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720dbff207a94212bdaf227d9b05d627" is_pre_election: true
I20260807 11:09:00.037487 25153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1924e8081c31446c8953300614a93d43" candidate_uuid: "3751e3c65b0340eeba13019e16fcc3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" is_pre_election: true
I20260807 11:09:00.037515 25152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1eee668b4cc4eddad42edaf1827309d" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" is_pre_election: true
I20260807 11:09:00.037998 25334 leader_election.cc:288] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:09:00.038287 25153 raft_consensus.cc:2454] T 1924e8081c31446c8953300614a93d43 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3751e3c65b0340eeba13019e16fcc3bf in term 0.
I20260807 11:09:00.037580 25047 raft_consensus.cc:2454] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3751e3c65b0340eeba13019e16fcc3bf in term 0.
I20260807 11:09:00.039628 25207 leader_election.cc:302] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, bbeb19c5189a46bcb4c327db2b6bc861; no voters:
I20260807 11:09:00.039683 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1eee668b4cc4eddad42edaf1827309d" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf" is_pre_election: true
I20260807 11:09:00.040462 25260 raft_consensus.cc:2454] T c1eee668b4cc4eddad42edaf1827309d P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 720dbff207a94212bdaf227d9b05d627 in term 0.
I20260807 11:09:00.040984 25390 raft_consensus.cc:2781] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:00.041390 25390 raft_consensus.cc:482] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:00.041566 24993 leader_election.cc:302] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, 720dbff207a94212bdaf227d9b05d627; no voters:
I20260807 11:09:00.041800 25390 raft_consensus.cc:3037] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:00.042439 25334 raft_consensus.cc:2781] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:00.042902 25334 raft_consensus.cc:482] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:00.042924 25368 raft_consensus.cc:504] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:00.043639 25334 raft_consensus.cc:3037] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:00.044307 25152 raft_consensus.cc:2379] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 720dbff207a94212bdaf227d9b05d627 in current term 1: Already voted for candidate bbeb19c5189a46bcb4c327db2b6bc861 in this term.
I20260807 11:09:00.045079 25368 leader_election.cc:288] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 1 election: Requested vote from peers 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:09:00.046979 25047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1eee668b4cc4eddad42edaf1827309d" candidate_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720dbff207a94212bdaf227d9b05d627"
I20260807 11:09:00.047118 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1eee668b4cc4eddad42edaf1827309d" candidate_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
I20260807 11:09:00.047775 25260 raft_consensus.cc:3037] T c1eee668b4cc4eddad42edaf1827309d P 3751e3c65b0340eeba13019e16fcc3bf [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:00.049139 25390 raft_consensus.cc:504] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:00.050613 25334 raft_consensus.cc:504] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:00.051616 25047 raft_consensus.cc:2379] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bbeb19c5189a46bcb4c327db2b6bc861 in current term 1: Already voted for candidate 720dbff207a94212bdaf227d9b05d627 in this term.
I20260807 11:09:00.052205 25390 leader_election.cc:288] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 1 election: Requested vote from peers 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989), bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315)
I20260807 11:09:00.052251 25048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1924e8081c31446c8953300614a93d43" candidate_uuid: "3751e3c65b0340eeba13019e16fcc3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720dbff207a94212bdaf227d9b05d627"
I20260807 11:09:00.052438 25152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1924e8081c31446c8953300614a93d43" candidate_uuid: "3751e3c65b0340eeba13019e16fcc3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
I20260807 11:09:00.053170 25048 raft_consensus.cc:3037] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:00.053889 25334 leader_election.cc:288] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 1 election: Requested vote from peers bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:09:00.054616 25152 raft_consensus.cc:3037] T 1924e8081c31446c8953300614a93d43 P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:00.055881 25260 raft_consensus.cc:2454] T c1eee668b4cc4eddad42edaf1827309d P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbeb19c5189a46bcb4c327db2b6bc861 in term 1.
I20260807 11:09:00.056025 25153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1eee668b4cc4eddad42edaf1827309d" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
I20260807 11:09:00.056854 25153 raft_consensus.cc:2379] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 720dbff207a94212bdaf227d9b05d627 in current term 1: Already voted for candidate bbeb19c5189a46bcb4c327db2b6bc861 in this term.
I20260807 11:09:00.057195 25095 leader_election.cc:302] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, bbeb19c5189a46bcb4c327db2b6bc861; no voters: 720dbff207a94212bdaf227d9b05d627
I20260807 11:09:00.057435 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1eee668b4cc4eddad42edaf1827309d" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
I20260807 11:09:00.058262 25260 raft_consensus.cc:2379] T c1eee668b4cc4eddad42edaf1827309d P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 720dbff207a94212bdaf227d9b05d627 in current term 1: Already voted for candidate bbeb19c5189a46bcb4c327db2b6bc861 in this term.
I20260807 11:09:00.059792 25368 raft_consensus.cc:2781] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:00.060312 25368 raft_consensus.cc:686] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 LEADER]: Becoming Leader. State: Replica: bbeb19c5189a46bcb4c327db2b6bc861, State: Running, Role: LEADER
I20260807 11:09:00.060540 25048 raft_consensus.cc:2454] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3751e3c65b0340eeba13019e16fcc3bf in term 1.
I20260807 11:09:00.061291 25368 consensus_queue.cc:237] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:00.062526 24990 leader_election.cc:302] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 720dbff207a94212bdaf227d9b05d627; no voters: 3751e3c65b0340eeba13019e16fcc3bf, bbeb19c5189a46bcb4c327db2b6bc861
I20260807 11:09:00.061676 25210 leader_election.cc:302] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, 720dbff207a94212bdaf227d9b05d627; no voters:
I20260807 11:09:00.063758 25152 raft_consensus.cc:2454] T 1924e8081c31446c8953300614a93d43 P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3751e3c65b0340eeba13019e16fcc3bf in term 1.
I20260807 11:09:00.063982 25334 raft_consensus.cc:2726] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260807 11:09:00.065140 25390 raft_consensus.cc:2781] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:00.066362 25390 raft_consensus.cc:686] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 LEADER]: Becoming Leader. State: Replica: 3751e3c65b0340eeba13019e16fcc3bf, State: Running, Role: LEADER
I20260807 11:09:00.067138 25390 consensus_queue.cc:237] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:00.072714 24899 catalog_manager.cc:5495] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130). New cstate: current_term: 1 leader_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:00.075129 25390 consensus_queue.cc:1035] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:00.076092 24900 catalog_manager.cc:5495] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131). New cstate: current_term: 1 leader_uuid: "3751e3c65b0340eeba13019e16fcc3bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } health_report { overall_health: HEALTHY } } }
I20260807 11:09:00.090119 25333 consensus_queue.cc:1035] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:09:00.105968 25368 raft_consensus.cc:482] T 89018784373e4903ae1a1bcaa1ac3f4e P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:00.106393 25368 raft_consensus.cc:504] T 89018784373e4903ae1a1bcaa1ac3f4e P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:00.108155 25368 leader_election.cc:288] T 89018784373e4903ae1a1bcaa1ac3f4e P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:09:00.108933 25048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89018784373e4903ae1a1bcaa1ac3f4e" candidate_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720dbff207a94212bdaf227d9b05d627" is_pre_election: true
I20260807 11:09:00.109323 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89018784373e4903ae1a1bcaa1ac3f4e" candidate_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf" is_pre_election: true
I20260807 11:09:00.109576 25048 raft_consensus.cc:2379] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbeb19c5189a46bcb4c327db2b6bc861 in current term 1: Already voted for candidate 3751e3c65b0340eeba13019e16fcc3bf in this term.
I20260807 11:09:00.110720 25095 leader_election.cc:302] T 89018784373e4903ae1a1bcaa1ac3f4e P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbeb19c5189a46bcb4c327db2b6bc861; no voters: 3751e3c65b0340eeba13019e16fcc3bf, 720dbff207a94212bdaf227d9b05d627
I20260807 11:09:00.111433 25368 raft_consensus.cc:3037] T 89018784373e4903ae1a1bcaa1ac3f4e P bbeb19c5189a46bcb4c327db2b6bc861 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:00.118005 25368 raft_consensus.cc:2726] T 89018784373e4903ae1a1bcaa1ac3f4e P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:00.155186 25334 consensus_queue.cc:1035] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:09:00.166463 25334 consensus_queue.cc:1035] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:09:00.178449 25368 consensus_queue.cc:1035] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:00.191762 25336 consensus_queue.cc:1035] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:09:00.263888 25333 consensus_queue.cc:1035] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:09:00.276896 25386 consensus_queue.cc:1035] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:09:00.351819 25260 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1924e8081c31446c8953300614a93d43"
dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
mode: GRACEFUL
new_leader_uuid: "720dbff207a94212bdaf227d9b05d627"
from {username='slave'} at 127.0.0.1:38128
I20260807 11:09:00.352380 25260 raft_consensus.cc:595] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 LEADER]: Received request to transfer leadership to TS 720dbff207a94212bdaf227d9b05d627
I20260807 11:09:00.356963 25260 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "89018784373e4903ae1a1bcaa1ac3f4e"
dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
mode: GRACEFUL
new_leader_uuid: "720dbff207a94212bdaf227d9b05d627"
from {username='slave'} at 127.0.0.1:38128
I20260807 11:09:00.357455 25260 raft_consensus.cc:595] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [term 1 LEADER]: Received request to transfer leadership to TS 720dbff207a94212bdaf227d9b05d627
I20260807 11:09:00.369390 25260 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a59d5a742d82444391f7103f701851bf"
dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
mode: GRACEFUL
new_leader_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
from {username='slave'} at 127.0.0.1:38128
I20260807 11:09:00.369884 25260 raft_consensus.cc:595] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [term 1 LEADER]: Received request to transfer leadership to TS bbeb19c5189a46bcb4c327db2b6bc861
I20260807 11:09:00.376873 25386 consensus_queue.cc:1035] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:09:00.377079 25260 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dfd144f74da942029b3904cc9cfda2a2"
dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
mode: GRACEFUL
new_leader_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
from {username='slave'} at 127.0.0.1:38128
I20260807 11:09:00.377686 25260 raft_consensus.cc:595] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 LEADER]: Received request to transfer leadership to TS bbeb19c5189a46bcb4c327db2b6bc861
I20260807 11:09:00.390054 25390 consensus_queue.cc:1035] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:00.405476 25390 raft_consensus.cc:979] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf: : Instructing follower bbeb19c5189a46bcb4c327db2b6bc861 to start an election
I20260807 11:09:00.405922 25390 raft_consensus.cc:1067] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [term 1 LEADER]: Signalling peer bbeb19c5189a46bcb4c327db2b6bc861 to start an election
I20260807 11:09:00.407495 25152 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a59d5a742d82444391f7103f701851bf"
dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
from {username='slave'} at 127.0.0.1:34964
I20260807 11:09:00.408037 25152 raft_consensus.cc:482] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:09:00.408413 25152 raft_consensus.cc:3037] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:00.410662 25153 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e25a8abd35e45c2af9bdf5f6b053269"
dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
mode: GRACEFUL
new_leader_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
from {username='slave'} at 127.0.0.1:34978
I20260807 11:09:00.411257 25153 raft_consensus.cc:595] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 LEADER]: Received request to transfer leadership to TS 3751e3c65b0340eeba13019e16fcc3bf
I20260807 11:09:00.414346 25152 raft_consensus.cc:504] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:00.415726 25152 leader_election.cc:288] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 2 election: Requested vote from peers 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:09:00.416663 25048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59d5a742d82444391f7103f701851bf" candidate_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "720dbff207a94212bdaf227d9b05d627"
I20260807 11:09:00.416926 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59d5a742d82444391f7103f701851bf" candidate_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
I20260807 11:09:00.417264 25048 raft_consensus.cc:3037] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:00.417469 25260 raft_consensus.cc:3032] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:09:00.417793 25260 raft_consensus.cc:729] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3751e3c65b0340eeba13019e16fcc3bf, State: Running, Role: LEADER
I20260807 11:09:00.418424 25260 consensus_queue.cc:260] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:00.419457 25260 raft_consensus.cc:3037] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:00.422241 25048 raft_consensus.cc:2454] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbeb19c5189a46bcb4c327db2b6bc861 in term 2.
I20260807 11:09:00.423255 25096 leader_election.cc:302] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 720dbff207a94212bdaf227d9b05d627, bbeb19c5189a46bcb4c327db2b6bc861; no voters:
I20260807 11:09:00.424045 25336 raft_consensus.cc:2781] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:09:00.424187 25260 raft_consensus.cc:2454] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbeb19c5189a46bcb4c327db2b6bc861 in term 2.
I20260807 11:09:00.424454 25336 raft_consensus.cc:686] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 LEADER]: Becoming Leader. State: Replica: bbeb19c5189a46bcb4c327db2b6bc861, State: Running, Role: LEADER
I20260807 11:09:00.425453 25336 consensus_queue.cc:237] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:00.431715 24900 catalog_manager.cc:5495] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 reported cstate change: term changed from 1 to 2, leader changed from 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131) to bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130). New cstate: current_term: 2 leader_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:00.478911 25370 raft_consensus.cc:979] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861: : Instructing follower 3751e3c65b0340eeba13019e16fcc3bf to start an election
I20260807 11:09:00.479416 25368 raft_consensus.cc:1067] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 LEADER]: Signalling peer 3751e3c65b0340eeba13019e16fcc3bf to start an election
I20260807 11:09:00.480993 25260 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e25a8abd35e45c2af9bdf5f6b053269"
dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
from {username='slave'} at 127.0.0.1:38118
I20260807 11:09:00.481547 25260 raft_consensus.cc:482] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:09:00.481473 25368 consensus_queue.cc:1035] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:09:00.481976 25260 raft_consensus.cc:3037] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:00.488588 25260 raft_consensus.cc:504] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:00.491469 25260 leader_election.cc:288] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 2 election: Requested vote from peers 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989), bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315)
I20260807 11:09:00.492897 25047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e25a8abd35e45c2af9bdf5f6b053269" candidate_uuid: "3751e3c65b0340eeba13019e16fcc3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "720dbff207a94212bdaf227d9b05d627"
I20260807 11:09:00.493502 25047 raft_consensus.cc:3037] T 5e25a8abd35e45c2af9bdf5f6b053269 P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:00.493638 25336 consensus_queue.cc:1035] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:00.501127 25047 raft_consensus.cc:2454] T 5e25a8abd35e45c2af9bdf5f6b053269 P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3751e3c65b0340eeba13019e16fcc3bf in term 2.
I20260807 11:09:00.503022 25210 leader_election.cc:302] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, 720dbff207a94212bdaf227d9b05d627; no voters:
I20260807 11:09:00.503912 25386 raft_consensus.cc:2781] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:09:00.504364 25386 raft_consensus.cc:686] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 LEADER]: Becoming Leader. State: Replica: 3751e3c65b0340eeba13019e16fcc3bf, State: Running, Role: LEADER
I20260807 11:09:00.504921 25152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e25a8abd35e45c2af9bdf5f6b053269" candidate_uuid: "3751e3c65b0340eeba13019e16fcc3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
I20260807 11:09:00.505534 25152 raft_consensus.cc:3032] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:09:00.505165 25386 consensus_queue.cc:237] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:00.505857 25152 raft_consensus.cc:729] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbeb19c5189a46bcb4c327db2b6bc861, State: Running, Role: LEADER
I20260807 11:09:00.506584 25152 consensus_queue.cc:260] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:00.508172 25152 raft_consensus.cc:3037] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:00.515061 25152 raft_consensus.cc:2454] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3751e3c65b0340eeba13019e16fcc3bf in term 2.
I20260807 11:09:00.515326 24900 catalog_manager.cc:5495] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf reported cstate change: term changed from 1 to 2, leader changed from bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130) to 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131). New cstate: current_term: 2 leader_uuid: "3751e3c65b0340eeba13019e16fcc3bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } health_report { overall_health: HEALTHY } } }
I20260807 11:09:00.536010 25386 consensus_queue.cc:1035] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:09:00.546480 25333 consensus_queue.cc:1035] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:09:00.553107 25333 raft_consensus.cc:979] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf: : Instructing follower 720dbff207a94212bdaf227d9b05d627 to start an election
I20260807 11:09:00.553546 25390 raft_consensus.cc:1067] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 LEADER]: Signalling peer 720dbff207a94212bdaf227d9b05d627 to start an election
I20260807 11:09:00.555122 25048 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1924e8081c31446c8953300614a93d43"
dest_uuid: "720dbff207a94212bdaf227d9b05d627"
from {username='slave'} at 127.0.0.1:35176
I20260807 11:09:00.555509 25048 raft_consensus.cc:482] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:09:00.555806 25048 raft_consensus.cc:3037] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:00.560384 25048 raft_consensus.cc:504] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:00.562310 25048 leader_election.cc:288] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 2 election: Requested vote from peers bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:09:00.562649 25152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1924e8081c31446c8953300614a93d43" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
I20260807 11:09:00.562990 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1924e8081c31446c8953300614a93d43" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
I20260807 11:09:00.563266 25152 raft_consensus.cc:3037] T 1924e8081c31446c8953300614a93d43 P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:00.563583 25260 raft_consensus.cc:3032] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:09:00.563956 25260 raft_consensus.cc:729] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3751e3c65b0340eeba13019e16fcc3bf, State: Running, Role: LEADER
I20260807 11:09:00.564648 25260 consensus_queue.cc:260] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:00.565670 25260 raft_consensus.cc:3037] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:00.569417 25152 raft_consensus.cc:2454] T 1924e8081c31446c8953300614a93d43 P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720dbff207a94212bdaf227d9b05d627 in term 2.
I20260807 11:09:00.570389 24990 leader_election.cc:302] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 720dbff207a94212bdaf227d9b05d627, bbeb19c5189a46bcb4c327db2b6bc861; no voters:
I20260807 11:09:00.570698 25260 raft_consensus.cc:2454] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720dbff207a94212bdaf227d9b05d627 in term 2.
I20260807 11:09:00.571327 25362 raft_consensus.cc:2781] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:09:00.571866 25362 raft_consensus.cc:686] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [term 2 LEADER]: Becoming Leader. State: Replica: 720dbff207a94212bdaf227d9b05d627, State: Running, Role: LEADER
I20260807 11:09:00.572628 25362 consensus_queue.cc:237] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:00.578734 24899 catalog_manager.cc:5495] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 reported cstate change: term changed from 1 to 2, leader changed from 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131) to 720dbff207a94212bdaf227d9b05d627 (127.23.120.129). New cstate: current_term: 2 leader_uuid: "720dbff207a94212bdaf227d9b05d627" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:00.583415 25333 raft_consensus.cc:979] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf: : Instructing follower bbeb19c5189a46bcb4c327db2b6bc861 to start an election
I20260807 11:09:00.583877 25390 raft_consensus.cc:1067] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 LEADER]: Signalling peer bbeb19c5189a46bcb4c327db2b6bc861 to start an election
I20260807 11:09:00.585417 25152 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dfd144f74da942029b3904cc9cfda2a2"
dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
from {username='slave'} at 127.0.0.1:34964
I20260807 11:09:00.585963 25152 raft_consensus.cc:482] T dfd144f74da942029b3904cc9cfda2a2 P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:09:00.586311 25152 raft_consensus.cc:3037] T dfd144f74da942029b3904cc9cfda2a2 P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:00.591421 25152 raft_consensus.cc:504] T dfd144f74da942029b3904cc9cfda2a2 P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:00.592903 25152 leader_election.cc:288] T dfd144f74da942029b3904cc9cfda2a2 P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 2 election: Requested vote from peers 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:09:00.593899 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfd144f74da942029b3904cc9cfda2a2" candidate_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
I20260807 11:09:00.593935 25048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfd144f74da942029b3904cc9cfda2a2" candidate_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "720dbff207a94212bdaf227d9b05d627"
I20260807 11:09:00.594503 25260 raft_consensus.cc:3032] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:09:00.594537 25048 raft_consensus.cc:3037] T dfd144f74da942029b3904cc9cfda2a2 P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:00.594858 25260 raft_consensus.cc:729] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3751e3c65b0340eeba13019e16fcc3bf, State: Running, Role: LEADER
I20260807 11:09:00.595566 25260 consensus_queue.cc:260] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:00.596451 25260 raft_consensus.cc:3037] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:00.599171 25048 raft_consensus.cc:2454] T dfd144f74da942029b3904cc9cfda2a2 P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbeb19c5189a46bcb4c327db2b6bc861 in term 2.
I20260807 11:09:00.600178 25096 leader_election.cc:302] T dfd144f74da942029b3904cc9cfda2a2 P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 720dbff207a94212bdaf227d9b05d627, bbeb19c5189a46bcb4c327db2b6bc861; no voters:
I20260807 11:09:00.600862 25336 raft_consensus.cc:2781] T dfd144f74da942029b3904cc9cfda2a2 P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:09:00.600917 25260 raft_consensus.cc:2454] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbeb19c5189a46bcb4c327db2b6bc861 in term 2.
I20260807 11:09:00.601388 25336 raft_consensus.cc:686] T dfd144f74da942029b3904cc9cfda2a2 P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 LEADER]: Becoming Leader. State: Replica: bbeb19c5189a46bcb4c327db2b6bc861, State: Running, Role: LEADER
I20260807 11:09:00.602216 25336 consensus_queue.cc:237] T dfd144f74da942029b3904cc9cfda2a2 P bbeb19c5189a46bcb4c327db2b6bc861 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:00.608400 24899 catalog_manager.cc:5495] T dfd144f74da942029b3904cc9cfda2a2 P bbeb19c5189a46bcb4c327db2b6bc861 reported cstate change: term changed from 1 to 2, leader changed from 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131) to bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130). New cstate: current_term: 2 leader_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:00.837203 25386 raft_consensus.cc:979] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf: : Instructing follower 720dbff207a94212bdaf227d9b05d627 to start an election
I20260807 11:09:00.837626 25390 raft_consensus.cc:1067] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [term 1 LEADER]: Signalling peer 720dbff207a94212bdaf227d9b05d627 to start an election
I20260807 11:09:00.838953 25048 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89018784373e4903ae1a1bcaa1ac3f4e"
dest_uuid: "720dbff207a94212bdaf227d9b05d627"
from {username='slave'} at 127.0.0.1:35176
I20260807 11:09:00.839438 25048 raft_consensus.cc:482] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:09:00.839768 25048 raft_consensus.cc:3037] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:00.844135 25048 raft_consensus.cc:504] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:00.845543 25048 leader_election.cc:288] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 2 election: Requested vote from peers bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:09:00.846418 25152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89018784373e4903ae1a1bcaa1ac3f4e" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
I20260807 11:09:00.846634 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89018784373e4903ae1a1bcaa1ac3f4e" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
I20260807 11:09:00.847018 25152 raft_consensus.cc:3037] T 89018784373e4903ae1a1bcaa1ac3f4e P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:00.847211 25260 raft_consensus.cc:3032] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:09:00.847569 25260 raft_consensus.cc:729] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3751e3c65b0340eeba13019e16fcc3bf, State: Running, Role: LEADER
I20260807 11:09:00.848868 25260 consensus_queue.cc:260] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:00.850948 25260 raft_consensus.cc:3037] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:00.853542 25152 raft_consensus.cc:2454] T 89018784373e4903ae1a1bcaa1ac3f4e P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720dbff207a94212bdaf227d9b05d627 in term 2.
I20260807 11:09:00.854508 24990 leader_election.cc:302] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 720dbff207a94212bdaf227d9b05d627, bbeb19c5189a46bcb4c327db2b6bc861; no voters:
I20260807 11:09:00.855158 25362 raft_consensus.cc:2781] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:09:00.855566 25362 raft_consensus.cc:686] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [term 2 LEADER]: Becoming Leader. State: Replica: 720dbff207a94212bdaf227d9b05d627, State: Running, Role: LEADER
I20260807 11:09:00.856444 25260 raft_consensus.cc:2454] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720dbff207a94212bdaf227d9b05d627 in term 2.
I20260807 11:09:00.856282 25362 consensus_queue.cc:237] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:00.862522 24899 catalog_manager.cc:5495] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 reported cstate change: term changed from 1 to 2, leader changed from 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131) to 720dbff207a94212bdaf227d9b05d627 (127.23.120.129). New cstate: current_term: 2 leader_uuid: "720dbff207a94212bdaf227d9b05d627" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:00.919082 25048 raft_consensus.cc:1261] T 5e25a8abd35e45c2af9bdf5f6b053269 P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Refusing update from remote peer 3751e3c65b0340eeba13019e16fcc3bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:09:00.920399 25390 consensus_queue.cc:1035] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:00.927572 25152 raft_consensus.cc:1261] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Refusing update from remote peer 3751e3c65b0340eeba13019e16fcc3bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:09:00.929363 25386 consensus_queue.cc:1035] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:00.964296 25152 raft_consensus.cc:1261] T 1924e8081c31446c8953300614a93d43 P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Refusing update from remote peer 720dbff207a94212bdaf227d9b05d627: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:09:00.965543 25362 consensus_queue.cc:1035] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:00.970150 25260 raft_consensus.cc:1261] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Refusing update from remote peer bbeb19c5189a46bcb4c327db2b6bc861: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:09:00.973294 25370 consensus_queue.cc:1035] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:00.975291 25260 raft_consensus.cc:1261] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Refusing update from remote peer 720dbff207a94212bdaf227d9b05d627: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:09:00.978542 25362 consensus_queue.cc:1035] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:00.986996 25048 raft_consensus.cc:1261] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Refusing update from remote peer bbeb19c5189a46bcb4c327db2b6bc861: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:09:00.988937 25370 consensus_queue.cc:1035] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:01.015174 25048 raft_consensus.cc:1261] T dfd144f74da942029b3904cc9cfda2a2 P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Refusing update from remote peer bbeb19c5189a46bcb4c327db2b6bc861: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:09:01.016307 25370 consensus_queue.cc:1035] T dfd144f74da942029b3904cc9cfda2a2 P bbeb19c5189a46bcb4c327db2b6bc861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:01.023662 25261 raft_consensus.cc:1261] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Refusing update from remote peer bbeb19c5189a46bcb4c327db2b6bc861: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:09:01.025838 25336 consensus_queue.cc:1035] T dfd144f74da942029b3904cc9cfda2a2 P bbeb19c5189a46bcb4c327db2b6bc861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:01.296939 25261 raft_consensus.cc:1261] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Refusing update from remote peer 720dbff207a94212bdaf227d9b05d627: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:09:01.297984 25362 consensus_queue.cc:1035] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:01.308322 25152 raft_consensus.cc:1261] T 89018784373e4903ae1a1bcaa1ac3f4e P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Refusing update from remote peer 720dbff207a94212bdaf227d9b05d627: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:09:01.309798 25362 consensus_queue.cc:1035] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:03.441781 24034 auto_leader_rebalancer-test.cc:339] The leader distribution is
I20260807 11:09:03.443017 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:09:03.462779 25048 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "425296c8f30a46e0a77af3752db337bf"
dest_uuid: "720dbff207a94212bdaf227d9b05d627"
mode: GRACEFUL
new_leader_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
from {username='slave'} at 127.0.0.1:35186
I20260807 11:09:03.463503 25048 raft_consensus.cc:595] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [term 1 LEADER]: Received request to transfer leadership to TS 3751e3c65b0340eeba13019e16fcc3bf
I20260807 11:09:03.466012 25048 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7e4964373b984d6089c2c83d13a52792"
dest_uuid: "720dbff207a94212bdaf227d9b05d627"
mode: GRACEFUL
new_leader_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
from {username='slave'} at 127.0.0.1:35186
I20260807 11:09:03.466468 25048 raft_consensus.cc:595] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [term 1 LEADER]: Received request to transfer leadership to TS 3751e3c65b0340eeba13019e16fcc3bf
I20260807 11:09:03.467406 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260807 11:09:03.467820 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:09:03.728413 25476 raft_consensus.cc:979] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627: : Instructing follower 3751e3c65b0340eeba13019e16fcc3bf to start an election
I20260807 11:09:03.729059 25470 raft_consensus.cc:1067] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [term 1 LEADER]: Signalling peer 3751e3c65b0340eeba13019e16fcc3bf to start an election
I20260807 11:09:03.730665 25261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "425296c8f30a46e0a77af3752db337bf"
dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
from {username='slave'} at 127.0.0.1:38104
I20260807 11:09:03.731246 25261 raft_consensus.cc:482] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:09:03.731602 25261 raft_consensus.cc:3037] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:03.738134 25261 raft_consensus.cc:504] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:03.740118 25261 leader_election.cc:288] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 2 election: Requested vote from peers 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989), bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315)
I20260807 11:09:03.741436 25048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425296c8f30a46e0a77af3752db337bf" candidate_uuid: "3751e3c65b0340eeba13019e16fcc3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "720dbff207a94212bdaf227d9b05d627"
I20260807 11:09:03.741478 25152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425296c8f30a46e0a77af3752db337bf" candidate_uuid: "3751e3c65b0340eeba13019e16fcc3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
I20260807 11:09:03.742138 25048 raft_consensus.cc:3032] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:09:03.742228 25152 raft_consensus.cc:3037] T 425296c8f30a46e0a77af3752db337bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:03.742381 25048 raft_consensus.cc:729] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 720dbff207a94212bdaf227d9b05d627, State: Running, Role: LEADER
I20260807 11:09:03.743230 25048 consensus_queue.cc:260] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:03.744287 25048 raft_consensus.cc:3037] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:03.748936 25152 raft_consensus.cc:2454] T 425296c8f30a46e0a77af3752db337bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3751e3c65b0340eeba13019e16fcc3bf in term 2.
I20260807 11:09:03.748930 25048 raft_consensus.cc:2454] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3751e3c65b0340eeba13019e16fcc3bf in term 2.
I20260807 11:09:03.750514 25210 leader_election.cc:302] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, 720dbff207a94212bdaf227d9b05d627, bbeb19c5189a46bcb4c327db2b6bc861; no voters:
I20260807 11:09:03.751376 25469 raft_consensus.cc:2781] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:09:03.752257 25469 raft_consensus.cc:686] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [term 2 LEADER]: Becoming Leader. State: Replica: 3751e3c65b0340eeba13019e16fcc3bf, State: Running, Role: LEADER
I20260807 11:09:03.753151 25469 consensus_queue.cc:237] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:03.760862 24899 catalog_manager.cc:5495] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf reported cstate change: term changed from 1 to 2, leader changed from 720dbff207a94212bdaf227d9b05d627 (127.23.120.129) to 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131). New cstate: current_term: 2 leader_uuid: "3751e3c65b0340eeba13019e16fcc3bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } health_report { overall_health: HEALTHY } } }
I20260807 11:09:03.798350 25476 raft_consensus.cc:979] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627: : Instructing follower 3751e3c65b0340eeba13019e16fcc3bf to start an election
I20260807 11:09:03.798812 25470 raft_consensus.cc:1067] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [term 1 LEADER]: Signalling peer 3751e3c65b0340eeba13019e16fcc3bf to start an election
I20260807 11:09:03.800163 25261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7e4964373b984d6089c2c83d13a52792"
dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
from {username='slave'} at 127.0.0.1:38104
I20260807 11:09:03.800669 25261 raft_consensus.cc:482] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:09:03.801040 25261 raft_consensus.cc:3037] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:03.807039 25261 raft_consensus.cc:504] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } } peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } }
I20260807 11:09:03.808508 25261 leader_election.cc:288] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 2 election: Requested vote from peers bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315), 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989)
I20260807 11:09:03.809166 25152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4964373b984d6089c2c83d13a52792" candidate_uuid: "3751e3c65b0340eeba13019e16fcc3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
I20260807 11:09:03.809571 25048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4964373b984d6089c2c83d13a52792" candidate_uuid: "3751e3c65b0340eeba13019e16fcc3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "720dbff207a94212bdaf227d9b05d627"
I20260807 11:09:03.809701 25152 raft_consensus.cc:3037] T 7e4964373b984d6089c2c83d13a52792 P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:03.810057 25048 raft_consensus.cc:3032] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:09:03.810268 25048 raft_consensus.cc:729] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 720dbff207a94212bdaf227d9b05d627, State: Running, Role: LEADER
I20260807 11:09:03.810921 25048 consensus_queue.cc:260] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } } peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } }
I20260807 11:09:03.812054 25048 raft_consensus.cc:3037] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:03.814530 25152 raft_consensus.cc:2454] T 7e4964373b984d6089c2c83d13a52792 P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3751e3c65b0340eeba13019e16fcc3bf in term 2.
I20260807 11:09:03.815584 25207 leader_election.cc:302] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, bbeb19c5189a46bcb4c327db2b6bc861; no voters:
I20260807 11:09:03.816325 25048 raft_consensus.cc:2454] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3751e3c65b0340eeba13019e16fcc3bf in term 2.
I20260807 11:09:03.816356 25469 raft_consensus.cc:2781] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:09:03.816982 25469 raft_consensus.cc:686] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 LEADER]: Becoming Leader. State: Replica: 3751e3c65b0340eeba13019e16fcc3bf, State: Running, Role: LEADER
I20260807 11:09:03.817764 25469 consensus_queue.cc:237] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } } peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } }
I20260807 11:09:03.825143 24899 catalog_manager.cc:5495] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf reported cstate change: term changed from 1 to 2, leader changed from 720dbff207a94212bdaf227d9b05d627 (127.23.120.129) to 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131). New cstate: current_term: 2 leader_uuid: "3751e3c65b0340eeba13019e16fcc3bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:04.137768 25048 raft_consensus.cc:1261] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Refusing update from remote peer 3751e3c65b0340eeba13019e16fcc3bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:09:04.139009 25469 consensus_queue.cc:1035] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:04.149631 25152 raft_consensus.cc:1261] T 425296c8f30a46e0a77af3752db337bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Refusing update from remote peer 3751e3c65b0340eeba13019e16fcc3bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:09:04.151485 25492 consensus_queue.cc:1035] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:09:04.204509 25048 raft_consensus.cc:1261] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Refusing update from remote peer 3751e3c65b0340eeba13019e16fcc3bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:09:04.205871 25469 consensus_queue.cc:1035] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:04.215219 25152 raft_consensus.cc:1261] T 7e4964373b984d6089c2c83d13a52792 P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Refusing update from remote peer 3751e3c65b0340eeba13019e16fcc3bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:09:04.217650 25498 consensus_queue.cc:1035] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:09:04.468809 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:09:04.477337 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:09:04.477674 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:09:05.478526 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:09:05.483906 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:09:05.484329 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:09:06.485220 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:09:06.491147 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:09:06.491593 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:09:07.492483 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:09:07.498135 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:09:07.498790 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:09:08.499564 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:09:08.504786 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:09:08.505215 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:09:09.505977 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:09:09.511204 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:09:09.511600 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:09:10.512327 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:09:10.518419 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:09:10.518899 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:09:11.519799 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:09:11.525803 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:09:11.526315 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:09:12.527289 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:09:12.533149 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:09:12.533613 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:09:13.564853 24034 auto_leader_rebalancer-test.cc:358] The leader distribution is
I20260807 11:09:13.569707 25261 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7e4964373b984d6089c2c83d13a52792"
dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
mode: GRACEFUL
new_leader_uuid: "720dbff207a94212bdaf227d9b05d627"
from {username='slave'} at 127.0.0.1:38128
I20260807 11:09:13.570333 25261 raft_consensus.cc:595] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 LEADER]: Received request to transfer leadership to TS 720dbff207a94212bdaf227d9b05d627
I20260807 11:09:13.579753 25261 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "425296c8f30a46e0a77af3752db337bf"
dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
mode: GRACEFUL
new_leader_uuid: "720dbff207a94212bdaf227d9b05d627"
from {username='slave'} at 127.0.0.1:38128
I20260807 11:09:13.580209 25261 raft_consensus.cc:595] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [term 2 LEADER]: Received request to transfer leadership to TS 720dbff207a94212bdaf227d9b05d627
I20260807 11:09:13.585347 25498 raft_consensus.cc:979] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf: : Instructing follower 720dbff207a94212bdaf227d9b05d627 to start an election
I20260807 11:09:13.586026 25516 raft_consensus.cc:1067] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 LEADER]: Signalling peer 720dbff207a94212bdaf227d9b05d627 to start an election
I20260807 11:09:13.587772 25048 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7e4964373b984d6089c2c83d13a52792"
dest_uuid: "720dbff207a94212bdaf227d9b05d627"
from {username='slave'} at 127.0.0.1:35176
I20260807 11:09:13.588284 25048 raft_consensus.cc:482] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:09:13.588699 25048 raft_consensus.cc:3037] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:09:13.588827 25153 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c1eee668b4cc4eddad42edaf1827309d"
dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
mode: GRACEFUL
new_leader_uuid: "720dbff207a94212bdaf227d9b05d627"
from {username='slave'} at 127.0.0.1:34978
I20260807 11:09:13.589422 25153 raft_consensus.cc:595] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 LEADER]: Received request to transfer leadership to TS 720dbff207a94212bdaf227d9b05d627
I20260807 11:09:13.594476 25153 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a59d5a742d82444391f7103f701851bf"
dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
mode: GRACEFUL
new_leader_uuid: "720dbff207a94212bdaf227d9b05d627"
from {username='slave'} at 127.0.0.1:34978
I20260807 11:09:13.594976 25153 raft_consensus.cc:595] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 LEADER]: Received request to transfer leadership to TS 720dbff207a94212bdaf227d9b05d627
I20260807 11:09:13.596046 25048 raft_consensus.cc:504] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } } peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } }
I20260807 11:09:13.598837 25048 leader_election.cc:288] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 3 election: Requested vote from peers bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:09:13.599645 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4964373b984d6089c2c83d13a52792" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
I20260807 11:09:13.600517 25261 raft_consensus.cc:3032] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:09:13.599704 25153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4964373b984d6089c2c83d13a52792" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
I20260807 11:09:13.600898 25261 raft_consensus.cc:729] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3751e3c65b0340eeba13019e16fcc3bf, State: Running, Role: LEADER
I20260807 11:09:13.600828 25152 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "39c732ca6d3a44758b2f1f46d56f9161"
dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
mode: GRACEFUL
new_leader_uuid: "720dbff207a94212bdaf227d9b05d627"
from {username='slave'} at 127.0.0.1:34978
I20260807 11:09:13.601547 25153 raft_consensus.cc:3037] T 7e4964373b984d6089c2c83d13a52792 P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:09:13.601794 25152 raft_consensus.cc:595] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 LEADER]: Received request to transfer leadership to TS 720dbff207a94212bdaf227d9b05d627
I20260807 11:09:13.602042 25261 consensus_queue.cc:260] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } } peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } }
I20260807 11:09:13.603283 25261 raft_consensus.cc:3037] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:09:13.610342 25153 raft_consensus.cc:2454] T 7e4964373b984d6089c2c83d13a52792 P bbeb19c5189a46bcb4c327db2b6bc861 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720dbff207a94212bdaf227d9b05d627 in term 3.
I20260807 11:09:13.610471 25261 raft_consensus.cc:2454] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720dbff207a94212bdaf227d9b05d627 in term 3.
I20260807 11:09:13.611662 24990 leader_election.cc:302] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 720dbff207a94212bdaf227d9b05d627, bbeb19c5189a46bcb4c327db2b6bc861; no voters:
I20260807 11:09:13.612495 25470 raft_consensus.cc:2781] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:09:13.613524 25470 raft_consensus.cc:686] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [term 3 LEADER]: Becoming Leader. State: Replica: 720dbff207a94212bdaf227d9b05d627, State: Running, Role: LEADER
I20260807 11:09:13.614341 25470 consensus_queue.cc:237] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } } peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } }
I20260807 11:09:13.622481 24898 catalog_manager.cc:5495] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 reported cstate change: term changed from 2 to 3, leader changed from 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131) to 720dbff207a94212bdaf227d9b05d627 (127.23.120.129). New cstate: current_term: 3 leader_uuid: "720dbff207a94212bdaf227d9b05d627" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } health_report { overall_health: HEALTHY } } }
I20260807 11:09:13.798602 25508 raft_consensus.cc:979] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861: : Instructing follower 720dbff207a94212bdaf227d9b05d627 to start an election
I20260807 11:09:13.799193 25529 raft_consensus.cc:1067] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 LEADER]: Signalling peer 720dbff207a94212bdaf227d9b05d627 to start an election
I20260807 11:09:13.800679 25048 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c1eee668b4cc4eddad42edaf1827309d"
dest_uuid: "720dbff207a94212bdaf227d9b05d627"
from {username='slave'} at 127.0.0.1:35164
I20260807 11:09:13.801182 25048 raft_consensus.cc:482] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:09:13.801477 25048 raft_consensus.cc:3037] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:13.806386 25048 raft_consensus.cc:504] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:13.807960 25048 leader_election.cc:288] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 2 election: Requested vote from peers bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:09:13.809006 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1eee668b4cc4eddad42edaf1827309d" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
I20260807 11:09:13.809067 25153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1eee668b4cc4eddad42edaf1827309d" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
I20260807 11:09:13.809468 25261 raft_consensus.cc:3037] T c1eee668b4cc4eddad42edaf1827309d P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:13.809713 25153 raft_consensus.cc:3032] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:09:13.810029 25153 raft_consensus.cc:729] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbeb19c5189a46bcb4c327db2b6bc861, State: Running, Role: LEADER
I20260807 11:09:13.810761 25153 consensus_queue.cc:260] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:13.811753 25153 raft_consensus.cc:3037] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:13.813738 25261 raft_consensus.cc:2454] T c1eee668b4cc4eddad42edaf1827309d P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720dbff207a94212bdaf227d9b05d627 in term 2.
I20260807 11:09:13.814786 24993 leader_election.cc:302] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, 720dbff207a94212bdaf227d9b05d627; no voters:
I20260807 11:09:13.815472 25470 raft_consensus.cc:2781] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:09:13.815834 25470 raft_consensus.cc:686] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [term 2 LEADER]: Becoming Leader. State: Replica: 720dbff207a94212bdaf227d9b05d627, State: Running, Role: LEADER
I20260807 11:09:13.816638 25470 consensus_queue.cc:237] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:13.817265 25153 raft_consensus.cc:2454] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720dbff207a94212bdaf227d9b05d627 in term 2.
I20260807 11:09:13.823442 24898 catalog_manager.cc:5495] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 reported cstate change: term changed from 1 to 2, leader changed from bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130) to 720dbff207a94212bdaf227d9b05d627 (127.23.120.129). New cstate: current_term: 2 leader_uuid: "720dbff207a94212bdaf227d9b05d627" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:13.865938 25498 raft_consensus.cc:979] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf: : Instructing follower 720dbff207a94212bdaf227d9b05d627 to start an election
I20260807 11:09:13.866427 25516 raft_consensus.cc:1067] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [term 2 LEADER]: Signalling peer 720dbff207a94212bdaf227d9b05d627 to start an election
I20260807 11:09:13.867660 25048 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "425296c8f30a46e0a77af3752db337bf"
dest_uuid: "720dbff207a94212bdaf227d9b05d627"
from {username='slave'} at 127.0.0.1:35176
I20260807 11:09:13.868151 25048 raft_consensus.cc:482] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:09:13.868427 25048 raft_consensus.cc:3037] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:09:13.873000 25048 raft_consensus.cc:504] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:13.874684 25048 leader_election.cc:288] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 3 election: Requested vote from peers bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:09:13.875469 25152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425296c8f30a46e0a77af3752db337bf" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
I20260807 11:09:13.876135 25152 raft_consensus.cc:3037] T 425296c8f30a46e0a77af3752db337bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:09:13.876273 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425296c8f30a46e0a77af3752db337bf" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
I20260807 11:09:13.876879 25261 raft_consensus.cc:3032] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:09:13.877183 25261 raft_consensus.cc:729] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3751e3c65b0340eeba13019e16fcc3bf, State: Running, Role: LEADER
I20260807 11:09:13.877897 25261 consensus_queue.cc:260] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:13.878803 25261 raft_consensus.cc:3037] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:09:13.881186 25152 raft_consensus.cc:2454] T 425296c8f30a46e0a77af3752db337bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720dbff207a94212bdaf227d9b05d627 in term 3.
I20260807 11:09:13.882511 24990 leader_election.cc:302] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 720dbff207a94212bdaf227d9b05d627, bbeb19c5189a46bcb4c327db2b6bc861; no voters:
I20260807 11:09:13.883133 25470 raft_consensus.cc:2781] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:09:13.883600 25470 raft_consensus.cc:686] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [term 3 LEADER]: Becoming Leader. State: Replica: 720dbff207a94212bdaf227d9b05d627, State: Running, Role: LEADER
I20260807 11:09:13.884115 25529 raft_consensus.cc:979] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861: : Instructing follower 720dbff207a94212bdaf227d9b05d627 to start an election
I20260807 11:09:13.884163 25261 raft_consensus.cc:2454] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720dbff207a94212bdaf227d9b05d627 in term 3.
I20260807 11:09:13.884661 25529 raft_consensus.cc:1067] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 LEADER]: Signalling peer 720dbff207a94212bdaf227d9b05d627 to start an election
I20260807 11:09:13.884397 25470 consensus_queue.cc:237] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:13.886070 25048 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a59d5a742d82444391f7103f701851bf"
dest_uuid: "720dbff207a94212bdaf227d9b05d627"
from {username='slave'} at 127.0.0.1:35164
I20260807 11:09:13.886471 25048 raft_consensus.cc:482] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:09:13.886754 25048 raft_consensus.cc:3037] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:09:13.891410 25048 raft_consensus.cc:504] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:13.891750 24898 catalog_manager.cc:5495] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 reported cstate change: term changed from 2 to 3, leader changed from 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131) to 720dbff207a94212bdaf227d9b05d627 (127.23.120.129). New cstate: current_term: 3 leader_uuid: "720dbff207a94212bdaf227d9b05d627" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:13.893699 25152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59d5a742d82444391f7103f701851bf" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
I20260807 11:09:13.894003 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59d5a742d82444391f7103f701851bf" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
I20260807 11:09:13.896018 25152 raft_consensus.cc:3032] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:09:13.896117 25261 raft_consensus.cc:3037] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:09:13.896323 25152 raft_consensus.cc:729] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bbeb19c5189a46bcb4c327db2b6bc861, State: Running, Role: LEADER
I20260807 11:09:13.894423 25048 leader_election.cc:288] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 3 election: Requested vote from peers bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:09:13.897102 25152 consensus_queue.cc:260] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:13.898651 25152 raft_consensus.cc:3037] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:09:13.901352 25261 raft_consensus.cc:2454] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720dbff207a94212bdaf227d9b05d627 in term 3.
I20260807 11:09:13.902292 24993 leader_election.cc:302] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, 720dbff207a94212bdaf227d9b05d627; no voters:
I20260807 11:09:13.903062 25470 raft_consensus.cc:2781] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:09:13.903488 25470 raft_consensus.cc:686] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627 [term 3 LEADER]: Becoming Leader. State: Replica: 720dbff207a94212bdaf227d9b05d627, State: Running, Role: LEADER
I20260807 11:09:13.904012 25152 raft_consensus.cc:2454] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720dbff207a94212bdaf227d9b05d627 in term 3.
I20260807 11:09:13.904368 25470 consensus_queue.cc:237] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:13.911063 24898 catalog_manager.cc:5495] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627 reported cstate change: term changed from 2 to 3, leader changed from bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130) to 720dbff207a94212bdaf227d9b05d627 (127.23.120.129). New cstate: current_term: 3 leader_uuid: "720dbff207a94212bdaf227d9b05d627" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:14.083113 25152 raft_consensus.cc:1261] T 7e4964373b984d6089c2c83d13a52792 P bbeb19c5189a46bcb4c327db2b6bc861 [term 3 FOLLOWER]: Refusing update from remote peer 720dbff207a94212bdaf227d9b05d627: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:09:14.084085 25470 consensus_queue.cc:1035] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:14.094388 25508 raft_consensus.cc:979] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861: : Instructing follower 720dbff207a94212bdaf227d9b05d627 to start an election
I20260807 11:09:14.095628 25261 raft_consensus.cc:1261] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [term 3 FOLLOWER]: Refusing update from remote peer 720dbff207a94212bdaf227d9b05d627: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:09:14.097790 25470 consensus_queue.cc:1035] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:14.105113 25539 raft_consensus.cc:1067] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 LEADER]: Signalling peer 720dbff207a94212bdaf227d9b05d627 to start an election
I20260807 11:09:14.107081 25048 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "39c732ca6d3a44758b2f1f46d56f9161"
dest_uuid: "720dbff207a94212bdaf227d9b05d627"
from {username='slave'} at 127.0.0.1:35164
I20260807 11:09:14.107702 25048 raft_consensus.cc:482] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:09:14.108254 25048 raft_consensus.cc:3037] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:14.115087 25048 raft_consensus.cc:504] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:14.117234 25048 leader_election.cc:288] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 2 election: Requested vote from peers bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:09:14.118680 25152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c732ca6d3a44758b2f1f46d56f9161" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
I20260807 11:09:14.119114 25152 raft_consensus.cc:3032] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:09:14.119335 25152 raft_consensus.cc:729] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbeb19c5189a46bcb4c327db2b6bc861, State: Running, Role: LEADER
I20260807 11:09:14.120057 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c732ca6d3a44758b2f1f46d56f9161" candidate_uuid: "720dbff207a94212bdaf227d9b05d627" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
I20260807 11:09:14.120030 25152 consensus_queue.cc:260] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:14.120859 25261 raft_consensus.cc:3037] T 39c732ca6d3a44758b2f1f46d56f9161 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:14.121443 25152 raft_consensus.cc:3037] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:14.125681 25152 raft_consensus.cc:2454] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720dbff207a94212bdaf227d9b05d627 in term 2.
I20260807 11:09:14.125710 25261 raft_consensus.cc:2454] T 39c732ca6d3a44758b2f1f46d56f9161 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720dbff207a94212bdaf227d9b05d627 in term 2.
I20260807 11:09:14.126634 24993 leader_election.cc:302] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, 720dbff207a94212bdaf227d9b05d627; no voters:
I20260807 11:09:14.127389 25470 raft_consensus.cc:2781] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:09:14.127818 25470 raft_consensus.cc:686] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [term 2 LEADER]: Becoming Leader. State: Replica: 720dbff207a94212bdaf227d9b05d627, State: Running, Role: LEADER
I20260807 11:09:14.128388 25470 consensus_queue.cc:237] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:14.135142 24898 catalog_manager.cc:5495] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 reported cstate change: term changed from 1 to 2, leader changed from bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130) to 720dbff207a94212bdaf227d9b05d627 (127.23.120.129). New cstate: current_term: 2 leader_uuid: "720dbff207a94212bdaf227d9b05d627" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:14.278615 25152 raft_consensus.cc:1261] T 425296c8f30a46e0a77af3752db337bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 3 FOLLOWER]: Refusing update from remote peer 720dbff207a94212bdaf227d9b05d627: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:09:14.279779 25470 consensus_queue.cc:1035] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:14.289283 25261 raft_consensus.cc:1261] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [term 3 FOLLOWER]: Refusing update from remote peer 720dbff207a94212bdaf227d9b05d627: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:09:14.291916 25470 consensus_queue.cc:1035] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:14.296597 25260 raft_consensus.cc:1261] T c1eee668b4cc4eddad42edaf1827309d P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Refusing update from remote peer 720dbff207a94212bdaf227d9b05d627: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:09:14.298010 25547 consensus_queue.cc:1035] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:09:14.310576 25152 raft_consensus.cc:1261] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Refusing update from remote peer 720dbff207a94212bdaf227d9b05d627: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:09:14.312690 25546 consensus_queue.cc:1035] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:09:14.369341 25260 raft_consensus.cc:1261] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [term 3 FOLLOWER]: Refusing update from remote peer 720dbff207a94212bdaf227d9b05d627: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:09:14.370602 25546 consensus_queue.cc:1035] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:14.379642 25152 raft_consensus.cc:1261] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 3 FOLLOWER]: Refusing update from remote peer 720dbff207a94212bdaf227d9b05d627: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:09:14.381098 25547 consensus_queue.cc:1035] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:14.573325 25152 raft_consensus.cc:1261] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Refusing update from remote peer 720dbff207a94212bdaf227d9b05d627: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:09:14.574496 25547 consensus_queue.cc:1035] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:14.583597 25260 raft_consensus.cc:1261] T 39c732ca6d3a44758b2f1f46d56f9161 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Refusing update from remote peer 720dbff207a94212bdaf227d9b05d627: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:09:14.585057 25517 consensus_queue.cc:1035] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:09:16.639936 24034 auto_leader_rebalancer-test.cc:373] The leader distribution is
I20260807 11:09:16.640690 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:09:16.649245 25048 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1924e8081c31446c8953300614a93d43"
dest_uuid: "720dbff207a94212bdaf227d9b05d627"
mode: GRACEFUL
new_leader_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
from {username='slave'} at 127.0.0.1:35186
I20260807 11:09:16.649796 25048 raft_consensus.cc:595] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [term 2 LEADER]: Received request to transfer leadership to TS 3751e3c65b0340eeba13019e16fcc3bf
I20260807 11:09:16.652387 25048 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "39c732ca6d3a44758b2f1f46d56f9161"
dest_uuid: "720dbff207a94212bdaf227d9b05d627"
mode: GRACEFUL
new_leader_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
from {username='slave'} at 127.0.0.1:35186
I20260807 11:09:16.652822 25048 raft_consensus.cc:595] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [term 2 LEADER]: Received request to transfer leadership to TS bbeb19c5189a46bcb4c327db2b6bc861
I20260807 11:09:16.654861 25048 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c1eee668b4cc4eddad42edaf1827309d"
dest_uuid: "720dbff207a94212bdaf227d9b05d627"
mode: GRACEFUL
new_leader_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
from {username='slave'} at 127.0.0.1:35186
I20260807 11:09:16.655256 25048 raft_consensus.cc:595] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [term 2 LEADER]: Received request to transfer leadership to TS bbeb19c5189a46bcb4c327db2b6bc861
I20260807 11:09:16.657625 25048 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fd44192986994fc086a65a73dea598d6"
dest_uuid: "720dbff207a94212bdaf227d9b05d627"
mode: GRACEFUL
new_leader_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
from {username='slave'} at 127.0.0.1:35186
I20260807 11:09:16.658041 25048 raft_consensus.cc:595] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [term 1 LEADER]: Received request to transfer leadership to TS 3751e3c65b0340eeba13019e16fcc3bf
I20260807 11:09:16.659010 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260807 11:09:16.659446 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:09:16.661208 25564 raft_consensus.cc:979] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627: : Instructing follower bbeb19c5189a46bcb4c327db2b6bc861 to start an election
I20260807 11:09:16.661654 25517 raft_consensus.cc:1067] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [term 2 LEADER]: Signalling peer bbeb19c5189a46bcb4c327db2b6bc861 to start an election
I20260807 11:09:16.662840 25152 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "39c732ca6d3a44758b2f1f46d56f9161"
dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
from {username='slave'} at 127.0.0.1:34952
I20260807 11:09:16.663236 25152 raft_consensus.cc:482] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:09:16.663491 25152 raft_consensus.cc:3037] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:09:16.669885 25152 raft_consensus.cc:504] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:16.672254 25152 leader_election.cc:288] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 3 election: Requested vote from peers 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:09:16.673317 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c732ca6d3a44758b2f1f46d56f9161" candidate_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
I20260807 11:09:16.673467 25048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c732ca6d3a44758b2f1f46d56f9161" candidate_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "720dbff207a94212bdaf227d9b05d627"
I20260807 11:09:16.673945 25260 raft_consensus.cc:3037] T 39c732ca6d3a44758b2f1f46d56f9161 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:09:16.673993 25048 raft_consensus.cc:3032] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:09:16.674430 25048 raft_consensus.cc:729] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 720dbff207a94212bdaf227d9b05d627, State: Running, Role: LEADER
I20260807 11:09:16.675092 25048 consensus_queue.cc:260] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:16.676071 25048 raft_consensus.cc:3037] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:09:16.679800 25260 raft_consensus.cc:2454] T 39c732ca6d3a44758b2f1f46d56f9161 P 3751e3c65b0340eeba13019e16fcc3bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbeb19c5189a46bcb4c327db2b6bc861 in term 3.
I20260807 11:09:16.680902 25048 raft_consensus.cc:2454] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbeb19c5189a46bcb4c327db2b6bc861 in term 3.
I20260807 11:09:16.680917 25095 leader_election.cc:302] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, bbeb19c5189a46bcb4c327db2b6bc861; no voters:
I20260807 11:09:16.681933 25563 raft_consensus.cc:2781] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:09:16.682911 25563 raft_consensus.cc:686] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [term 3 LEADER]: Becoming Leader. State: Replica: bbeb19c5189a46bcb4c327db2b6bc861, State: Running, Role: LEADER
I20260807 11:09:16.683694 25563 consensus_queue.cc:237] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:16.690511 24898 catalog_manager.cc:5495] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 reported cstate change: term changed from 2 to 3, leader changed from 720dbff207a94212bdaf227d9b05d627 (127.23.120.129) to bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130). New cstate: current_term: 3 leader_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:16.697187 25564 raft_consensus.cc:979] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627: : Instructing follower bbeb19c5189a46bcb4c327db2b6bc861 to start an election
I20260807 11:09:16.697649 25517 raft_consensus.cc:1067] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [term 2 LEADER]: Signalling peer bbeb19c5189a46bcb4c327db2b6bc861 to start an election
I20260807 11:09:16.699129 25152 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c1eee668b4cc4eddad42edaf1827309d"
dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
from {username='slave'} at 127.0.0.1:34952
I20260807 11:09:16.699679 25152 raft_consensus.cc:482] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:09:16.700021 25152 raft_consensus.cc:3037] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:09:16.705111 25152 raft_consensus.cc:504] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:16.706475 25152 leader_election.cc:288] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 3 election: Requested vote from peers 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989), 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131:45885)
I20260807 11:09:16.707398 25048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1eee668b4cc4eddad42edaf1827309d" candidate_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "720dbff207a94212bdaf227d9b05d627"
I20260807 11:09:16.707405 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1eee668b4cc4eddad42edaf1827309d" candidate_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
I20260807 11:09:16.708178 25048 raft_consensus.cc:3032] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:09:16.708263 25260 raft_consensus.cc:3037] T c1eee668b4cc4eddad42edaf1827309d P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:09:16.708556 25048 raft_consensus.cc:729] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 720dbff207a94212bdaf227d9b05d627, State: Running, Role: LEADER
I20260807 11:09:16.709339 25048 consensus_queue.cc:260] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:16.710489 25048 raft_consensus.cc:3037] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:09:16.714560 25260 raft_consensus.cc:2454] T c1eee668b4cc4eddad42edaf1827309d P 3751e3c65b0340eeba13019e16fcc3bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbeb19c5189a46bcb4c327db2b6bc861 in term 3.
I20260807 11:09:16.715559 25095 leader_election.cc:302] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, bbeb19c5189a46bcb4c327db2b6bc861; no voters:
I20260807 11:09:16.716270 25563 raft_consensus.cc:2781] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:09:16.716674 25048 raft_consensus.cc:2454] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbeb19c5189a46bcb4c327db2b6bc861 in term 3.
I20260807 11:09:16.716683 25563 raft_consensus.cc:686] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 3 LEADER]: Becoming Leader. State: Replica: bbeb19c5189a46bcb4c327db2b6bc861, State: Running, Role: LEADER
I20260807 11:09:16.717594 25563 consensus_queue.cc:237] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:16.724478 24898 catalog_manager.cc:5495] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 reported cstate change: term changed from 2 to 3, leader changed from 720dbff207a94212bdaf227d9b05d627 (127.23.120.129) to bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130). New cstate: current_term: 3 leader_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:17.013010 25564 raft_consensus.cc:979] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627: : Instructing follower 3751e3c65b0340eeba13019e16fcc3bf to start an election
I20260807 11:09:17.013442 25517 raft_consensus.cc:1067] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [term 2 LEADER]: Signalling peer 3751e3c65b0340eeba13019e16fcc3bf to start an election
I20260807 11:09:17.014856 25260 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1924e8081c31446c8953300614a93d43"
dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
from {username='slave'} at 127.0.0.1:38104
I20260807 11:09:17.015283 25260 raft_consensus.cc:482] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:09:17.015514 25260 raft_consensus.cc:3037] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:09:17.019771 25260 raft_consensus.cc:504] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:17.021533 25260 leader_election.cc:288] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 3 election: Requested vote from peers 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989), bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315)
I20260807 11:09:17.022378 25048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1924e8081c31446c8953300614a93d43" candidate_uuid: "3751e3c65b0340eeba13019e16fcc3bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "720dbff207a94212bdaf227d9b05d627"
I20260807 11:09:17.022701 25152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1924e8081c31446c8953300614a93d43" candidate_uuid: "3751e3c65b0340eeba13019e16fcc3bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
I20260807 11:09:17.022948 25048 raft_consensus.cc:3032] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:09:17.023213 25152 raft_consensus.cc:3037] T 1924e8081c31446c8953300614a93d43 P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:09:17.023283 25048 raft_consensus.cc:729] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 720dbff207a94212bdaf227d9b05d627, State: Running, Role: LEADER
I20260807 11:09:17.024009 25048 consensus_queue.cc:260] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:17.025163 25048 raft_consensus.cc:3037] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:09:17.029407 25152 raft_consensus.cc:2454] T 1924e8081c31446c8953300614a93d43 P bbeb19c5189a46bcb4c327db2b6bc861 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3751e3c65b0340eeba13019e16fcc3bf in term 3.
I20260807 11:09:17.030646 25207 leader_election.cc:302] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, bbeb19c5189a46bcb4c327db2b6bc861; no voters:
I20260807 11:09:17.031014 25048 raft_consensus.cc:2454] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3751e3c65b0340eeba13019e16fcc3bf in term 3.
I20260807 11:09:17.031576 25560 raft_consensus.cc:2781] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:09:17.032841 25560 raft_consensus.cc:686] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [term 3 LEADER]: Becoming Leader. State: Replica: 3751e3c65b0340eeba13019e16fcc3bf, State: Running, Role: LEADER
I20260807 11:09:17.033833 25560 consensus_queue.cc:237] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:17.041117 24898 catalog_manager.cc:5495] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf reported cstate change: term changed from 2 to 3, leader changed from 720dbff207a94212bdaf227d9b05d627 (127.23.120.129) to 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131). New cstate: current_term: 3 leader_uuid: "3751e3c65b0340eeba13019e16fcc3bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } health_report { overall_health: HEALTHY } } }
I20260807 11:09:17.081245 25260 raft_consensus.cc:1261] T 39c732ca6d3a44758b2f1f46d56f9161 P 3751e3c65b0340eeba13019e16fcc3bf [term 3 FOLLOWER]: Refusing update from remote peer bbeb19c5189a46bcb4c327db2b6bc861: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:09:17.082404 25563 consensus_queue.cc:1035] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:17.092072 25048 raft_consensus.cc:1261] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [term 3 FOLLOWER]: Refusing update from remote peer bbeb19c5189a46bcb4c327db2b6bc861: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:09:17.093403 25565 consensus_queue.cc:1035] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:17.145816 25048 raft_consensus.cc:1261] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [term 3 FOLLOWER]: Refusing update from remote peer bbeb19c5189a46bcb4c327db2b6bc861: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:09:17.147197 25565 consensus_queue.cc:1035] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:17.155489 25260 raft_consensus.cc:1261] T c1eee668b4cc4eddad42edaf1827309d P 3751e3c65b0340eeba13019e16fcc3bf [term 3 FOLLOWER]: Refusing update from remote peer bbeb19c5189a46bcb4c327db2b6bc861: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:09:17.157197 25563 consensus_queue.cc:1035] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:17.221442 25564 raft_consensus.cc:979] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627: : Instructing follower 3751e3c65b0340eeba13019e16fcc3bf to start an election
I20260807 11:09:17.221850 25564 raft_consensus.cc:1067] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [term 1 LEADER]: Signalling peer 3751e3c65b0340eeba13019e16fcc3bf to start an election
I20260807 11:09:17.223194 25261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fd44192986994fc086a65a73dea598d6"
dest_uuid: "3751e3c65b0340eeba13019e16fcc3bf"
from {username='slave'} at 127.0.0.1:38104
I20260807 11:09:17.223644 25261 raft_consensus.cc:482] T fd44192986994fc086a65a73dea598d6 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:09:17.223867 25261 raft_consensus.cc:3037] T fd44192986994fc086a65a73dea598d6 P 3751e3c65b0340eeba13019e16fcc3bf [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:17.229283 25261 raft_consensus.cc:504] T fd44192986994fc086a65a73dea598d6 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:17.230798 25261 leader_election.cc:288] T fd44192986994fc086a65a73dea598d6 P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 2 election: Requested vote from peers 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989), bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315)
I20260807 11:09:17.231583 25048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd44192986994fc086a65a73dea598d6" candidate_uuid: "3751e3c65b0340eeba13019e16fcc3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "720dbff207a94212bdaf227d9b05d627"
I20260807 11:09:17.231840 25152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd44192986994fc086a65a73dea598d6" candidate_uuid: "3751e3c65b0340eeba13019e16fcc3bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bbeb19c5189a46bcb4c327db2b6bc861"
I20260807 11:09:17.232131 25048 raft_consensus.cc:3032] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:09:17.232419 25152 raft_consensus.cc:3037] T fd44192986994fc086a65a73dea598d6 P bbeb19c5189a46bcb4c327db2b6bc861 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:17.232441 25048 raft_consensus.cc:729] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 720dbff207a94212bdaf227d9b05d627, State: Running, Role: LEADER
I20260807 11:09:17.233376 25048 consensus_queue.cc:260] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:17.234652 25048 raft_consensus.cc:3037] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:17.237779 25152 raft_consensus.cc:2454] T fd44192986994fc086a65a73dea598d6 P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3751e3c65b0340eeba13019e16fcc3bf in term 2.
I20260807 11:09:17.238692 25207 leader_election.cc:302] T fd44192986994fc086a65a73dea598d6 P 3751e3c65b0340eeba13019e16fcc3bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3751e3c65b0340eeba13019e16fcc3bf, bbeb19c5189a46bcb4c327db2b6bc861; no voters:
I20260807 11:09:17.239065 25048 raft_consensus.cc:2454] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3751e3c65b0340eeba13019e16fcc3bf in term 2.
I20260807 11:09:17.239396 25560 raft_consensus.cc:2781] T fd44192986994fc086a65a73dea598d6 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:09:17.239867 25560 raft_consensus.cc:686] T fd44192986994fc086a65a73dea598d6 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 LEADER]: Becoming Leader. State: Replica: 3751e3c65b0340eeba13019e16fcc3bf, State: Running, Role: LEADER
I20260807 11:09:17.240586 25560 consensus_queue.cc:237] T fd44192986994fc086a65a73dea598d6 P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } }
I20260807 11:09:17.246888 24898 catalog_manager.cc:5495] T fd44192986994fc086a65a73dea598d6 P 3751e3c65b0340eeba13019e16fcc3bf reported cstate change: term changed from 1 to 2, leader changed from 720dbff207a94212bdaf227d9b05d627 (127.23.120.129) to 3751e3c65b0340eeba13019e16fcc3bf (127.23.120.131). New cstate: current_term: 2 leader_uuid: "3751e3c65b0340eeba13019e16fcc3bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3751e3c65b0340eeba13019e16fcc3bf" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 45885 } health_report { overall_health: HEALTHY } } }
I20260807 11:09:17.556974 25048 raft_consensus.cc:1261] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [term 3 FOLLOWER]: Refusing update from remote peer 3751e3c65b0340eeba13019e16fcc3bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:09:17.558147 25560 consensus_queue.cc:1035] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:17.570024 25152 raft_consensus.cc:1261] T 1924e8081c31446c8953300614a93d43 P bbeb19c5189a46bcb4c327db2b6bc861 [term 3 FOLLOWER]: Refusing update from remote peer 3751e3c65b0340eeba13019e16fcc3bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:09:17.571489 25575 consensus_queue.cc:1035] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:09:17.660310 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:09:17.666393 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:09:17.666723 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:09:17.788895 25048 raft_consensus.cc:1261] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Refusing update from remote peer 3751e3c65b0340eeba13019e16fcc3bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:09:17.790071 25575 consensus_queue.cc:1035] T fd44192986994fc086a65a73dea598d6 P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "720dbff207a94212bdaf227d9b05d627" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:17.798488 25152 raft_consensus.cc:1261] T fd44192986994fc086a65a73dea598d6 P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Refusing update from remote peer 3751e3c65b0340eeba13019e16fcc3bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:09:17.799835 25560 consensus_queue.cc:1035] T fd44192986994fc086a65a73dea598d6 P 3751e3c65b0340eeba13019e16fcc3bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbeb19c5189a46bcb4c327db2b6bc861" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:18.667577 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:09:18.674074 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:09:18.674476 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:09:19.675516 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:09:19.682042 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:09:19.682510 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:09:20.683274 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:09:20.688660 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:09:20.689098 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:09:21.690083 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:09:21.696421 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:09:21.696851 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:09:22.697574 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:09:22.703938 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:09:22.704320 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:09:23.705365 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:09:23.711930 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:09:23.712369 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:09:24.713290 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:09:24.719578 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:09:24.720019 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:09:25.721179 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:09:25.726260 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 11:09:25.726704 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:09:26.764287 24034 auto_leader_rebalancer-test.cc:384] The leader distribution is
I20260807 11:09:26.765071 24034 tablet_server.cc:178] TabletServer@127.23.120.129:0 shutting down...
I20260807 11:09:26.801909 24034 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260807 11:09:26.802711 24034 tablet_replica.cc:330] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627: stopping tablet replica
I20260807 11:09:26.803495 24034 raft_consensus.cc:2229] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627 [term 3 LEADER]: Raft consensus shutting down.
I20260807 11:09:26.804420 24034 raft_consensus.cc:2258] T a59d5a742d82444391f7103f701851bf P 720dbff207a94212bdaf227d9b05d627 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:26.806463 24034 tablet_replica.cc:330] T 5e25a8abd35e45c2af9bdf5f6b053269 P 720dbff207a94212bdaf227d9b05d627: stopping tablet replica
I20260807 11:09:26.806958 24034 raft_consensus.cc:2229] T 5e25a8abd35e45c2af9bdf5f6b053269 P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:26.807341 24034 raft_consensus.cc:2258] T 5e25a8abd35e45c2af9bdf5f6b053269 P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:26.809085 24034 tablet_replica.cc:330] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627: stopping tablet replica
I20260807 11:09:26.809559 24034 raft_consensus.cc:2229] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:26.809932 24034 raft_consensus.cc:2258] T 1924e8081c31446c8953300614a93d43 P 720dbff207a94212bdaf227d9b05d627 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:26.811532 24034 tablet_replica.cc:330] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627: stopping tablet replica
I20260807 11:09:26.811990 24034 raft_consensus.cc:2229] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:26.812475 24034 raft_consensus.cc:2258] T fd44192986994fc086a65a73dea598d6 P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:26.814313 24034 tablet_replica.cc:330] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627: stopping tablet replica
I20260807 11:09:26.814934 24034 raft_consensus.cc:2229] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:26.815442 24034 raft_consensus.cc:2258] T c1eee668b4cc4eddad42edaf1827309d P 720dbff207a94212bdaf227d9b05d627 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:26.818118 24034 tablet_replica.cc:330] T dfd144f74da942029b3904cc9cfda2a2 P 720dbff207a94212bdaf227d9b05d627: stopping tablet replica
I20260807 11:09:26.818658 24034 raft_consensus.cc:2229] T dfd144f74da942029b3904cc9cfda2a2 P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:26.819092 24034 raft_consensus.cc:2258] T dfd144f74da942029b3904cc9cfda2a2 P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:26.821000 24034 tablet_replica.cc:330] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627: stopping tablet replica
I20260807 11:09:26.821537 24034 raft_consensus.cc:2229] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [term 2 LEADER]: Raft consensus shutting down.
I20260807 11:09:26.822324 24034 raft_consensus.cc:2258] T 89018784373e4903ae1a1bcaa1ac3f4e P 720dbff207a94212bdaf227d9b05d627 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:26.824159 24034 tablet_replica.cc:330] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627: stopping tablet replica
I20260807 11:09:26.824599 24034 raft_consensus.cc:2229] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [term 3 LEADER]: Raft consensus shutting down.
I20260807 11:09:26.825403 24034 raft_consensus.cc:2258] T 425296c8f30a46e0a77af3752db337bf P 720dbff207a94212bdaf227d9b05d627 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:26.827072 24034 tablet_replica.cc:330] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627: stopping tablet replica
I20260807 11:09:26.827551 24034 raft_consensus.cc:2229] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:26.827936 24034 raft_consensus.cc:2258] T 39c732ca6d3a44758b2f1f46d56f9161 P 720dbff207a94212bdaf227d9b05d627 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:26.829635 24034 tablet_replica.cc:330] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627: stopping tablet replica
I20260807 11:09:26.830035 24034 raft_consensus.cc:2229] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [term 3 LEADER]: Raft consensus shutting down.
I20260807 11:09:26.830797 24034 raft_consensus.cc:2258] T 7e4964373b984d6089c2c83d13a52792 P 720dbff207a94212bdaf227d9b05d627 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260807 11:09:26.836501 25096 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 4 similar messages]
W20260807 11:09:26.839920 25096 consensus_peers.cc:487] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 -> Peer 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989): Couldn't send request to peer 720dbff207a94212bdaf227d9b05d627. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:26.863418 24034 tablet_server.cc:194] TabletServer@127.23.120.129:0 shutdown complete.
W20260807 11:09:26.879575 25210 consensus_peers.cc:487] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf -> Peer 720dbff207a94212bdaf227d9b05d627 (127.23.120.129:37989): Couldn't send request to peer 720dbff207a94212bdaf227d9b05d627. Status: Network error: Client connection negotiation failed: client connection to 127.23.120.129:37989: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:26.887249 24034 tablet_server.cc:178] TabletServer@127.23.120.130:0 shutting down...
I20260807 11:09:26.914949 24034 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260807 11:09:26.915551 24034 tablet_replica.cc:330] T 89018784373e4903ae1a1bcaa1ac3f4e P bbeb19c5189a46bcb4c327db2b6bc861: stopping tablet replica
I20260807 11:09:26.916342 24034 raft_consensus.cc:2229] T 89018784373e4903ae1a1bcaa1ac3f4e P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:26.916932 24034 raft_consensus.cc:2258] T 89018784373e4903ae1a1bcaa1ac3f4e P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:26.919421 24034 tablet_replica.cc:330] T 425296c8f30a46e0a77af3752db337bf P bbeb19c5189a46bcb4c327db2b6bc861: stopping tablet replica
I20260807 11:09:26.919946 24034 raft_consensus.cc:2229] T 425296c8f30a46e0a77af3752db337bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:26.920399 24034 raft_consensus.cc:2258] T 425296c8f30a46e0a77af3752db337bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:26.922106 24034 tablet_replica.cc:330] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861: stopping tablet replica
I20260807 11:09:26.922617 24034 raft_consensus.cc:2229] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 3 LEADER]: Raft consensus shutting down.
I20260807 11:09:26.923429 24034 raft_consensus.cc:2258] T c1eee668b4cc4eddad42edaf1827309d P bbeb19c5189a46bcb4c327db2b6bc861 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:26.925195 24034 tablet_replica.cc:330] T 1924e8081c31446c8953300614a93d43 P bbeb19c5189a46bcb4c327db2b6bc861: stopping tablet replica
I20260807 11:09:26.925726 24034 raft_consensus.cc:2229] T 1924e8081c31446c8953300614a93d43 P bbeb19c5189a46bcb4c327db2b6bc861 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:26.926162 24034 raft_consensus.cc:2258] T 1924e8081c31446c8953300614a93d43 P bbeb19c5189a46bcb4c327db2b6bc861 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:26.927784 24034 tablet_replica.cc:330] T fd44192986994fc086a65a73dea598d6 P bbeb19c5189a46bcb4c327db2b6bc861: stopping tablet replica
I20260807 11:09:26.928257 24034 raft_consensus.cc:2229] T fd44192986994fc086a65a73dea598d6 P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:26.928644 24034 raft_consensus.cc:2258] T fd44192986994fc086a65a73dea598d6 P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:26.930208 24034 tablet_replica.cc:330] T dfd144f74da942029b3904cc9cfda2a2 P bbeb19c5189a46bcb4c327db2b6bc861: stopping tablet replica
I20260807 11:09:26.930684 24034 raft_consensus.cc:2229] T dfd144f74da942029b3904cc9cfda2a2 P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 LEADER]: Raft consensus shutting down.
I20260807 11:09:26.931455 24034 raft_consensus.cc:2258] T dfd144f74da942029b3904cc9cfda2a2 P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:26.933161 24034 tablet_replica.cc:330] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861: stopping tablet replica
I20260807 11:09:26.933624 24034 raft_consensus.cc:2229] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:26.934041 24034 raft_consensus.cc:2258] T 5e25a8abd35e45c2af9bdf5f6b053269 P bbeb19c5189a46bcb4c327db2b6bc861 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:26.936265 24034 tablet_replica.cc:330] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861: stopping tablet replica
I20260807 11:09:26.936787 24034 raft_consensus.cc:2229] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [term 3 LEADER]: Raft consensus shutting down.
W20260807 11:09:26.937095 25207 consensus_peers.cc:487] T fd44192986994fc086a65a73dea598d6 P 3751e3c65b0340eeba13019e16fcc3bf -> Peer bbeb19c5189a46bcb4c327db2b6bc861 (127.23.120.130:44315): Couldn't send request to peer bbeb19c5189a46bcb4c327db2b6bc861. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:26.937683 24034 raft_consensus.cc:2258] T 39c732ca6d3a44758b2f1f46d56f9161 P bbeb19c5189a46bcb4c327db2b6bc861 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:26.939204 24034 tablet_replica.cc:330] T 7e4964373b984d6089c2c83d13a52792 P bbeb19c5189a46bcb4c327db2b6bc861: stopping tablet replica
I20260807 11:09:26.939601 24034 raft_consensus.cc:2229] T 7e4964373b984d6089c2c83d13a52792 P bbeb19c5189a46bcb4c327db2b6bc861 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:26.939970 24034 raft_consensus.cc:2258] T 7e4964373b984d6089c2c83d13a52792 P bbeb19c5189a46bcb4c327db2b6bc861 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:26.941500 24034 tablet_replica.cc:330] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861: stopping tablet replica
I20260807 11:09:26.941879 24034 raft_consensus.cc:2229] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:26.942202 24034 raft_consensus.cc:2258] T a59d5a742d82444391f7103f701851bf P bbeb19c5189a46bcb4c327db2b6bc861 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:26.972013 24034 tablet_server.cc:194] TabletServer@127.23.120.130:0 shutdown complete.
I20260807 11:09:26.990952 24034 tablet_server.cc:178] TabletServer@127.23.120.131:0 shutting down...
I20260807 11:09:27.019002 24034 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260807 11:09:27.019851 24034 tablet_replica.cc:330] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf: stopping tablet replica
I20260807 11:09:27.020711 24034 raft_consensus.cc:2229] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [term 3 LEADER]: Raft consensus shutting down.
I20260807 11:09:27.021682 24034 raft_consensus.cc:2258] T 1924e8081c31446c8953300614a93d43 P 3751e3c65b0340eeba13019e16fcc3bf [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:27.023810 24034 tablet_replica.cc:330] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf: stopping tablet replica
I20260807 11:09:27.024338 24034 raft_consensus.cc:2229] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:27.024890 24034 raft_consensus.cc:2258] T a59d5a742d82444391f7103f701851bf P 3751e3c65b0340eeba13019e16fcc3bf [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:27.026597 24034 tablet_replica.cc:330] T c1eee668b4cc4eddad42edaf1827309d P 3751e3c65b0340eeba13019e16fcc3bf: stopping tablet replica
I20260807 11:09:27.027122 24034 raft_consensus.cc:2229] T c1eee668b4cc4eddad42edaf1827309d P 3751e3c65b0340eeba13019e16fcc3bf [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:27.027573 24034 raft_consensus.cc:2258] T c1eee668b4cc4eddad42edaf1827309d P 3751e3c65b0340eeba13019e16fcc3bf [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:27.029213 24034 tablet_replica.cc:330] T fd44192986994fc086a65a73dea598d6 P 3751e3c65b0340eeba13019e16fcc3bf: stopping tablet replica
I20260807 11:09:27.029717 24034 raft_consensus.cc:2229] T fd44192986994fc086a65a73dea598d6 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 LEADER]: Raft consensus shutting down.
I20260807 11:09:27.030484 24034 raft_consensus.cc:2258] T fd44192986994fc086a65a73dea598d6 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:27.032214 24034 tablet_replica.cc:330] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf: stopping tablet replica
I20260807 11:09:27.032696 24034 raft_consensus.cc:2229] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 LEADER]: Raft consensus shutting down.
I20260807 11:09:27.033514 24034 raft_consensus.cc:2258] T 5e25a8abd35e45c2af9bdf5f6b053269 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:27.035076 24034 tablet_replica.cc:330] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf: stopping tablet replica
I20260807 11:09:27.035543 24034 raft_consensus.cc:2229] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:27.035969 24034 raft_consensus.cc:2258] T 89018784373e4903ae1a1bcaa1ac3f4e P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:27.037667 24034 tablet_replica.cc:330] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf: stopping tablet replica
I20260807 11:09:27.038131 24034 raft_consensus.cc:2229] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:27.038573 24034 raft_consensus.cc:2258] T 425296c8f30a46e0a77af3752db337bf P 3751e3c65b0340eeba13019e16fcc3bf [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:27.040108 24034 tablet_replica.cc:330] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf: stopping tablet replica
I20260807 11:09:27.040565 24034 raft_consensus.cc:2229] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:27.041006 24034 raft_consensus.cc:2258] T dfd144f74da942029b3904cc9cfda2a2 P 3751e3c65b0340eeba13019e16fcc3bf [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:27.042551 24034 tablet_replica.cc:330] T 39c732ca6d3a44758b2f1f46d56f9161 P 3751e3c65b0340eeba13019e16fcc3bf: stopping tablet replica
I20260807 11:09:27.043011 24034 raft_consensus.cc:2229] T 39c732ca6d3a44758b2f1f46d56f9161 P 3751e3c65b0340eeba13019e16fcc3bf [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:27.043385 24034 raft_consensus.cc:2258] T 39c732ca6d3a44758b2f1f46d56f9161 P 3751e3c65b0340eeba13019e16fcc3bf [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:27.044952 24034 tablet_replica.cc:330] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf: stopping tablet replica
I20260807 11:09:27.045379 24034 raft_consensus.cc:2229] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:27.045780 24034 raft_consensus.cc:2258] T 7e4964373b984d6089c2c83d13a52792 P 3751e3c65b0340eeba13019e16fcc3bf [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:27.075474 24034 tablet_server.cc:194] TabletServer@127.23.120.131:0 shutdown complete.
I20260807 11:09:27.094950 24034 master.cc:537] Master@127.23.120.190:46611 shutting down...
I20260807 11:09:27.119330 24034 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2e2233d81e9e472a90a0fb6f42174060 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:09:27.119889 24034 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2e2233d81e9e472a90a0fb6f42174060 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:27.120177 24034 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2e2233d81e9e472a90a0fb6f42174060: stopping tablet replica
I20260807 11:09:27.138447 24034 master.cc:552] Master@127.23.120.190:46611 shutdown complete.
3751e3c65b0340eeba13019e16fcc3bf 1
720dbff207a94212bdaf227d9b05d627 5
bbeb19c5189a46bcb4c327db2b6bc861 4
3751e3c65b0340eeba13019e16fcc3bf 3
720dbff207a94212bdaf227d9b05d627 3
bbeb19c5189a46bcb4c327db2b6bc861 4
3751e3c65b0340eeba13019e16fcc3bf 1
720dbff207a94212bdaf227d9b05d627 8
bbeb19c5189a46bcb4c327db2b6bc861 1
3751e3c65b0340eeba13019e16fcc3bf 3
720dbff207a94212bdaf227d9b05d627 4
bbeb19c5189a46bcb4c327db2b6bc861 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28854 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260807 11:09:27.185899 24034 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.120.190:37321
I20260807 11:09:27.187124 24034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:09:27.193734 25624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:09:27.193832 25626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:09:27.194515 25630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:09:27.195215 24034 server_base.cc:1034] running on GCE node
I20260807 11:09:27.196202 24034 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 11:09:27.196383 24034 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 11:09:27.196568 24034 hybrid_clock.cc:648] HybridClock initialized: now 1786100967196537 us; error 0 us; skew 500 ppm
I20260807 11:09:27.197114 24034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:09:27.203279 24034 webserver.cc:460] Webserver started at http://127.23.120.190:40361/ using document root <none> and password file <none>
I20260807 11:09:27.203756 24034 fs_manager.cc:359] Metadata directory not provided
I20260807 11:09:27.203941 24034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:09:27.204245 24034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:09:27.205520 24034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/master-0-root/instance:
uuid: "98664bdefe2a4a829847688b8d0d50c1"
format_stamp: "Formatted at 2026-08-07 11:09:27 on dist-test-slave-g170"
I20260807 11:09:27.209939 24034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 11:09:27.213246 25636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:09:27.214030 24034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 11:09:27.214313 24034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/master-0-root
uuid: "98664bdefe2a4a829847688b8d0d50c1"
format_stamp: "Formatted at 2026-08-07 11:09:27 on dist-test-slave-g170"
I20260807 11:09:27.214596 24034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:09:27.234549 24034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:09:27.235807 24034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:09:27.275645 24034 rpc_server.cc:307] RPC server started. Bound to: 127.23.120.190:37321
I20260807 11:09:27.276062 25705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.120.190:37321 every 8 connection(s)
I20260807 11:09:27.279914 25706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:27.290791 25706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98664bdefe2a4a829847688b8d0d50c1: Bootstrap starting.
I20260807 11:09:27.295627 25706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98664bdefe2a4a829847688b8d0d50c1: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:27.300307 25706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98664bdefe2a4a829847688b8d0d50c1: No bootstrap required, opened a new log
I20260807 11:09:27.302639 25706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98664bdefe2a4a829847688b8d0d50c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98664bdefe2a4a829847688b8d0d50c1" member_type: VOTER }
I20260807 11:09:27.303194 25706 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98664bdefe2a4a829847688b8d0d50c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:27.303454 25706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98664bdefe2a4a829847688b8d0d50c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98664bdefe2a4a829847688b8d0d50c1, State: Initialized, Role: FOLLOWER
I20260807 11:09:27.304147 25706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98664bdefe2a4a829847688b8d0d50c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98664bdefe2a4a829847688b8d0d50c1" member_type: VOTER }
I20260807 11:09:27.304665 25706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98664bdefe2a4a829847688b8d0d50c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:09:27.305006 25706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98664bdefe2a4a829847688b8d0d50c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:09:27.305339 25706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98664bdefe2a4a829847688b8d0d50c1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:27.309746 25706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98664bdefe2a4a829847688b8d0d50c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98664bdefe2a4a829847688b8d0d50c1" member_type: VOTER }
I20260807 11:09:27.310295 25706 leader_election.cc:302] T 00000000000000000000000000000000 P 98664bdefe2a4a829847688b8d0d50c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98664bdefe2a4a829847688b8d0d50c1; no voters:
I20260807 11:09:27.311496 25706 leader_election.cc:288] T 00000000000000000000000000000000 P 98664bdefe2a4a829847688b8d0d50c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:09:27.311935 25710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98664bdefe2a4a829847688b8d0d50c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:27.313418 25710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98664bdefe2a4a829847688b8d0d50c1 [term 1 LEADER]: Becoming Leader. State: Replica: 98664bdefe2a4a829847688b8d0d50c1, State: Running, Role: LEADER
I20260807 11:09:27.314155 25710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98664bdefe2a4a829847688b8d0d50c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98664bdefe2a4a829847688b8d0d50c1" member_type: VOTER }
I20260807 11:09:27.314879 25706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98664bdefe2a4a829847688b8d0d50c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:09:27.319907 25712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98664bdefe2a4a829847688b8d0d50c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98664bdefe2a4a829847688b8d0d50c1. Latest consensus state: current_term: 1 leader_uuid: "98664bdefe2a4a829847688b8d0d50c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98664bdefe2a4a829847688b8d0d50c1" member_type: VOTER } }
I20260807 11:09:27.320245 25711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98664bdefe2a4a829847688b8d0d50c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98664bdefe2a4a829847688b8d0d50c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98664bdefe2a4a829847688b8d0d50c1" member_type: VOTER } }
I20260807 11:09:27.320835 25712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98664bdefe2a4a829847688b8d0d50c1 [sys.catalog]: This master's current role is: LEADER
I20260807 11:09:27.321086 25711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98664bdefe2a4a829847688b8d0d50c1 [sys.catalog]: This master's current role is: LEADER
I20260807 11:09:27.324890 25719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:09:27.333822 25719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:09:27.334929 24034 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260807 11:09:27.342883 25719 catalog_manager.cc:1347] Generated new cluster ID: 57e1dc6feb58471bb67f27021f2b3320
I20260807 11:09:27.343161 25719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:09:27.360334 25719 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:09:27.361913 25719 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:09:27.372890 25719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98664bdefe2a4a829847688b8d0d50c1: Generated new TSK 0
I20260807 11:09:27.373523 25719 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:09:27.401418 24034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:09:27.407513 25736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:09:27.408808 25737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:09:27.411012 25739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:09:27.411207 24034 server_base.cc:1034] running on GCE node
I20260807 11:09:27.412207 24034 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 11:09:27.412439 24034 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 11:09:27.412613 24034 hybrid_clock.cc:648] HybridClock initialized: now 1786100967412593 us; error 0 us; skew 500 ppm
I20260807 11:09:27.413162 24034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:09:27.415720 24034 webserver.cc:460] Webserver started at http://127.23.120.129:46757/ using document root <none> and password file <none>
I20260807 11:09:27.416282 24034 fs_manager.cc:359] Metadata directory not provided
I20260807 11:09:27.416501 24034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:09:27.416880 24034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:09:27.417975 24034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-0-root/instance:
uuid: "606d49e053a346e486f48cb31832b815"
format_stamp: "Formatted at 2026-08-07 11:09:27 on dist-test-slave-g170"
I20260807 11:09:27.422526 24034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 11:09:27.425714 25747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:09:27.426553 24034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 11:09:27.426945 24034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-0-root
uuid: "606d49e053a346e486f48cb31832b815"
format_stamp: "Formatted at 2026-08-07 11:09:27 on dist-test-slave-g170"
I20260807 11:09:27.427345 24034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:09:27.467485 24034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:09:27.468647 24034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:09:27.470162 24034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:09:27.472406 24034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:09:27.472615 24034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:09:27.472896 24034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:09:27.473094 24034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:09:27.514411 24034 rpc_server.cc:307] RPC server started. Bound to: 127.23.120.129:40195
I20260807 11:09:27.514523 25850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.120.129:40195 every 8 connection(s)
I20260807 11:09:27.520433 24034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:09:27.528553 25856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:09:27.529647 25857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:09:27.532289 24034 server_base.cc:1034] running on GCE node
W20260807 11:09:27.532749 25863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:09:27.533797 24034 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 11:09:27.534017 24034 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 11:09:27.534211 24034 hybrid_clock.cc:648] HybridClock initialized: now 1786100967534196 us; error 0 us; skew 500 ppm
I20260807 11:09:27.534763 24034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:09:27.537290 25851 heartbeater.cc:347] Connected to a master server at 127.23.120.190:37321
I20260807 11:09:27.537665 25851 heartbeater.cc:464] Registering TS with master...
I20260807 11:09:27.537883 24034 webserver.cc:460] Webserver started at http://127.23.120.130:35935/ using document root <none> and password file <none>
I20260807 11:09:27.538384 25851 heartbeater.cc:511] Master 127.23.120.190:37321 requested a full tablet report, sending...
I20260807 11:09:27.538444 24034 fs_manager.cc:359] Metadata directory not provided
I20260807 11:09:27.538829 24034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:09:27.539108 24034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:09:27.540485 24034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-1-root/instance:
uuid: "4fd6381544574ddca559ce63b9243efc"
format_stamp: "Formatted at 2026-08-07 11:09:27 on dist-test-slave-g170"
I20260807 11:09:27.540544 25660 ts_manager.cc:194] Registered new tserver with Master: 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195)
I20260807 11:09:27.542503 25660 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40714
I20260807 11:09:27.545686 24034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 11:09:27.549189 25868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:09:27.549939 24034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 11:09:27.550230 24034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-1-root
uuid: "4fd6381544574ddca559ce63b9243efc"
format_stamp: "Formatted at 2026-08-07 11:09:27 on dist-test-slave-g170"
I20260807 11:09:27.550529 24034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:09:27.575436 24034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:09:27.576728 24034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:09:27.578251 24034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:09:27.580484 24034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:09:27.580698 24034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:09:27.580963 24034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:09:27.581130 24034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:09:27.620263 24034 rpc_server.cc:307] RPC server started. Bound to: 127.23.120.130:36465
I20260807 11:09:27.620417 25964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.120.130:36465 every 8 connection(s)
I20260807 11:09:27.624924 24034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:09:27.632690 25968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:09:27.633411 25969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:09:27.638427 24034 server_base.cc:1034] running on GCE node
W20260807 11:09:27.639477 25971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:09:27.639742 25965 heartbeater.cc:347] Connected to a master server at 127.23.120.190:37321
I20260807 11:09:27.640208 25965 heartbeater.cc:464] Registering TS with master...
I20260807 11:09:27.640609 24034 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 11:09:27.640918 24034 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 11:09:27.641104 24034 hybrid_clock.cc:648] HybridClock initialized: now 1786100967641073 us; error 0 us; skew 500 ppm
I20260807 11:09:27.641036 25965 heartbeater.cc:511] Master 127.23.120.190:37321 requested a full tablet report, sending...
I20260807 11:09:27.641850 24034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:09:27.643236 25660 ts_manager.cc:194] Registered new tserver with Master: 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:27.644896 25660 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40716
I20260807 11:09:27.645118 24034 webserver.cc:460] Webserver started at http://127.23.120.131:35237/ using document root <none> and password file <none>
I20260807 11:09:27.645853 24034 fs_manager.cc:359] Metadata directory not provided
I20260807 11:09:27.646147 24034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:09:27.646450 24034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:09:27.647878 24034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-2-root/instance:
uuid: "b22d69db45194908ad3d345eba9badd9"
format_stamp: "Formatted at 2026-08-07 11:09:27 on dist-test-slave-g170"
I20260807 11:09:27.653050 24034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.007s
I20260807 11:09:27.656534 25980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:09:27.657528 24034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 11:09:27.657855 24034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-2-root
uuid: "b22d69db45194908ad3d345eba9badd9"
format_stamp: "Formatted at 2026-08-07 11:09:27 on dist-test-slave-g170"
I20260807 11:09:27.658159 24034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:09:27.694519 24034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:09:27.695793 24034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:09:27.697328 24034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:09:27.699740 24034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:09:27.699997 24034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:09:27.700248 24034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:09:27.700451 24034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:09:27.744230 24034 rpc_server.cc:307] RPC server started. Bound to: 127.23.120.131:46009
I20260807 11:09:27.744374 26083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.120.131:46009 every 8 connection(s)
I20260807 11:09:27.759944 26084 heartbeater.cc:347] Connected to a master server at 127.23.120.190:37321
I20260807 11:09:27.760318 26084 heartbeater.cc:464] Registering TS with master...
I20260807 11:09:27.760952 26084 heartbeater.cc:511] Master 127.23.120.190:37321 requested a full tablet report, sending...
I20260807 11:09:27.762850 25660 ts_manager.cc:194] Registered new tserver with Master: b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:27.763618 24034 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016091036s
I20260807 11:09:27.764815 25660 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40730
I20260807 11:09:27.766418 24034 test_util.cc:268] Using random seed: 163019133
I20260807 11:09:27.791239 25660 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40742:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:09:27.797389 25660 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:09:28.014174 26026 tablet_service.cc:1467] Processing CreateTablet for tablet cf86a21774c34f33a6540d1dad0de0c4 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260807 11:09:28.015723 26026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf86a21774c34f33a6540d1dad0de0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.016659 26032 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6408e9acee46adacd2bbd81669f27a (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260807 11:09:28.016989 25915 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6408e9acee46adacd2bbd81669f27a (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260807 11:09:28.018219 26032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c6408e9acee46adacd2bbd81669f27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.018355 25915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c6408e9acee46adacd2bbd81669f27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.021067 26019 tablet_service.cc:1467] Processing CreateTablet for tablet cb81660c5a8b4365a928d2ef1e4d0be5 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260807 11:09:28.022073 25910 tablet_service.cc:1467] Processing CreateTablet for tablet 409fcb387861467396e84e2b22b8f9be (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260807 11:09:28.022995 26011 tablet_service.cc:1467] Processing CreateTablet for tablet b8330847089e41b0811e28dc30529234 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260807 11:09:28.023622 25910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 409fcb387861467396e84e2b22b8f9be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.024475 26011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8330847089e41b0811e28dc30529234. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.024703 26018 tablet_service.cc:1467] Processing CreateTablet for tablet 84f78f4141b342d9b7e397096f34af59 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260807 11:09:28.025977 26018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84f78f4141b342d9b7e397096f34af59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.026098 25909 tablet_service.cc:1467] Processing CreateTablet for tablet 0e15f851c3b34d9893d8a5c89e0f98ff (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260807 11:09:28.027601 25909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e15f851c3b34d9893d8a5c89e0f98ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.029187 26015 tablet_service.cc:1467] Processing CreateTablet for tablet f75da3017fd744719647332808daba49 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260807 11:09:28.030473 26015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75da3017fd744719647332808daba49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.032035 25906 tablet_service.cc:1467] Processing CreateTablet for tablet 84f78f4141b342d9b7e397096f34af59 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260807 11:09:28.033715 26016 tablet_service.cc:1467] Processing CreateTablet for tablet cbfca1b0bf4c479a834433a4c0ed66ac (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260807 11:09:28.033914 25913 tablet_service.cc:1467] Processing CreateTablet for tablet cf86a21774c34f33a6540d1dad0de0c4 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260807 11:09:28.034984 26016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbfca1b0bf4c479a834433a4c0ed66ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.035477 25913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf86a21774c34f33a6540d1dad0de0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.030018 25907 tablet_service.cc:1467] Processing CreateTablet for tablet cb81660c5a8b4365a928d2ef1e4d0be5 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260807 11:09:28.017891 26025 tablet_service.cc:1467] Processing CreateTablet for tablet 409fcb387861467396e84e2b22b8f9be (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260807 11:09:28.037061 26024 tablet_service.cc:1467] Processing CreateTablet for tablet 0e15f851c3b34d9893d8a5c89e0f98ff (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260807 11:09:28.023620 26014 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff0e05310b145358043bba0a9a8ba11 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260807 11:09:28.040195 25904 tablet_service.cc:1467] Processing CreateTablet for tablet cbfca1b0bf4c479a834433a4c0ed66ac (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260807 11:09:28.042727 26019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb81660c5a8b4365a928d2ef1e4d0be5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.046113 25906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84f78f4141b342d9b7e397096f34af59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.046468 26024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e15f851c3b34d9893d8a5c89e0f98ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.052117 26014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff0e05310b145358043bba0a9a8ba11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.057147 25903 tablet_service.cc:1467] Processing CreateTablet for tablet f75da3017fd744719647332808daba49 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260807 11:09:28.058534 25903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75da3017fd744719647332808daba49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.061977 25901 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff0e05310b145358043bba0a9a8ba11 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260807 11:09:28.063365 25901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff0e05310b145358043bba0a9a8ba11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.065330 25900 tablet_service.cc:1467] Processing CreateTablet for tablet b8330847089e41b0811e28dc30529234 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260807 11:09:28.066624 25900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8330847089e41b0811e28dc30529234. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.071671 25910 tablet_service.cc:1467] Processing CreateTablet for tablet 85d06c4d618c4aa6af896f216680dd68 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260807 11:09:28.073837 26025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 409fcb387861467396e84e2b22b8f9be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.075098 25907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb81660c5a8b4365a928d2ef1e4d0be5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.077836 25904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbfca1b0bf4c479a834433a4c0ed66ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.078076 26024 tablet_service.cc:1467] Processing CreateTablet for tablet 85d06c4d618c4aa6af896f216680dd68 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260807 11:09:28.079291 26024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85d06c4d618c4aa6af896f216680dd68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.089609 25910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85d06c4d618c4aa6af896f216680dd68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.104358 26024 tablet_service.cc:1467] Processing CreateTablet for tablet 986fe968b7be4eab9c480ec85c0384b0 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260807 11:09:28.106587 26025 tablet_service.cc:1467] Processing CreateTablet for tablet 54a291aa09434b06adbb86586dacec0e (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260807 11:09:28.112069 26024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 986fe968b7be4eab9c480ec85c0384b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.113124 26025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54a291aa09434b06adbb86586dacec0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.125142 25785 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6408e9acee46adacd2bbd81669f27a (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260807 11:09:28.126881 25785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c6408e9acee46adacd2bbd81669f27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.136230 26024 tablet_service.cc:1467] Processing CreateTablet for tablet 76dbf7480b434c0ba6a902c817cf5455 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260807 11:09:28.144695 26024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76dbf7480b434c0ba6a902c817cf5455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.148592 26025 tablet_service.cc:1467] Processing CreateTablet for tablet 3395e44a980a4ac2bc07304ae2525b35 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260807 11:09:28.153431 26025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3395e44a980a4ac2bc07304ae2525b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.158766 25783 tablet_service.cc:1467] Processing CreateTablet for tablet cf86a21774c34f33a6540d1dad0de0c4 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260807 11:09:28.160517 25783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf86a21774c34f33a6540d1dad0de0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.204599 26024 tablet_service.cc:1467] Processing CreateTablet for tablet e702bb04fc98457f9d8ee704fcaeafcf (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260807 11:09:28.204751 26019 tablet_service.cc:1467] Processing CreateTablet for tablet 93d531084c9447b78cffb286d6b966b7 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260807 11:09:28.206600 26024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e702bb04fc98457f9d8ee704fcaeafcf. 1 dirs total, 0 dirs full, 0 dirs failed
W20260807 11:09:28.254441 25639 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 50143.7us
I20260807 11:09:28.261123 26019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d531084c9447b78cffb286d6b966b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.265118 25782 tablet_service.cc:1467] Processing CreateTablet for tablet 409fcb387861467396e84e2b22b8f9be (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260807 11:09:28.266494 25782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 409fcb387861467396e84e2b22b8f9be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.277884 26109 tablet_bootstrap.cc:492] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:28.282392 26026 tablet_service.cc:1467] Processing CreateTablet for tablet f5e378b8a19447d69c6c9433f071086e (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260807 11:09:28.283684 26025 tablet_service.cc:1467] Processing CreateTablet for tablet f35a17e1683e41f4b146284f5cf09c58 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260807 11:09:28.284485 26026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e378b8a19447d69c6c9433f071086e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.285316 26025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f35a17e1683e41f4b146284f5cf09c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.286437 26016 tablet_service.cc:1467] Processing CreateTablet for tablet b7c44ad3ac214b2d83c41bc5aa01f349 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260807 11:09:28.287626 26016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7c44ad3ac214b2d83c41bc5aa01f349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.290449 26015 tablet_service.cc:1467] Processing CreateTablet for tablet a09a74fdae5d44c8a9b375e1cfdea1e9 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260807 11:09:28.292059 26015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a09a74fdae5d44c8a9b375e1cfdea1e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.294150 26018 tablet_service.cc:1467] Processing CreateTablet for tablet a805d50b18af4d5eb99f7c96e790e1dd (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260807 11:09:28.295624 26018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a805d50b18af4d5eb99f7c96e790e1dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.296366 26032 tablet_service.cc:1467] Processing CreateTablet for tablet 3c69ebea3aeb47d48ab8e083fdf71f7c (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260807 11:09:28.298137 25906 tablet_service.cc:1467] Processing CreateTablet for tablet 54a291aa09434b06adbb86586dacec0e (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260807 11:09:28.308105 25906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54a291aa09434b06adbb86586dacec0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.309779 26108 tablet_bootstrap.cc:492] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:28.311354 25915 tablet_service.cc:1467] Processing CreateTablet for tablet 76dbf7480b434c0ba6a902c817cf5455 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260807 11:09:28.311650 26019 tablet_service.cc:1467] Processing CreateTablet for tablet 51fb619c2ada475f817fbcb9739a401a (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260807 11:09:28.312683 25915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76dbf7480b434c0ba6a902c817cf5455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.312988 26019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51fb619c2ada475f817fbcb9739a401a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.313735 26011 tablet_service.cc:1467] Processing CreateTablet for tablet 955db4a12dc04bfe8124047fc46823d3 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260807 11:09:28.314937 26011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 955db4a12dc04bfe8124047fc46823d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.308101 26024 tablet_service.cc:1467] Processing CreateTablet for tablet e0766ccf18b84e41ac68eb6599a834c5 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260807 11:09:28.316179 26024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0766ccf18b84e41ac68eb6599a834c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.320561 25900 tablet_service.cc:1467] Processing CreateTablet for tablet 3395e44a980a4ac2bc07304ae2525b35 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260807 11:09:28.321808 25900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3395e44a980a4ac2bc07304ae2525b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.327157 25778 tablet_service.cc:1467] Processing CreateTablet for tablet 84f78f4141b342d9b7e397096f34af59 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260807 11:09:28.327592 25777 tablet_service.cc:1467] Processing CreateTablet for tablet cbfca1b0bf4c479a834433a4c0ed66ac (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260807 11:09:28.328460 25778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84f78f4141b342d9b7e397096f34af59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.328311 25779 tablet_service.cc:1467] Processing CreateTablet for tablet cb81660c5a8b4365a928d2ef1e4d0be5 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260807 11:09:28.329629 25779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb81660c5a8b4365a928d2ef1e4d0be5. 1 dirs total, 0 dirs full, 0 dirs failed
W20260807 11:09:28.330946 25723 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 11:09:28.331594 25777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbfca1b0bf4c479a834433a4c0ed66ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.326459 25780 tablet_service.cc:1467] Processing CreateTablet for tablet 0e15f851c3b34d9893d8a5c89e0f98ff (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260807 11:09:28.333854 25780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e15f851c3b34d9893d8a5c89e0f98ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.342365 25776 tablet_service.cc:1467] Processing CreateTablet for tablet f75da3017fd744719647332808daba49 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260807 11:09:28.343699 25776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75da3017fd744719647332808daba49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.360435 25775 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff0e05310b145358043bba0a9a8ba11 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260807 11:09:28.362881 25774 tablet_service.cc:1467] Processing CreateTablet for tablet b8330847089e41b0811e28dc30529234 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260807 11:09:28.364113 26026 tablet_service.cc:1467] Processing CreateTablet for tablet f40bb008c149418989fdcd7c1e67d554 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260807 11:09:28.368017 26026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f40bb008c149418989fdcd7c1e67d554. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.368988 26025 tablet_service.cc:1467] Processing CreateTablet for tablet 8467fdc1886745c1a7049de84b30bcc0 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260807 11:09:28.373281 26025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8467fdc1886745c1a7049de84b30bcc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.369850 26018 tablet_service.cc:1467] Processing CreateTablet for tablet 66d10b6b12404f0488b9f83c8080069f (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260807 11:09:28.370493 26015 tablet_service.cc:1467] Processing CreateTablet for tablet a3d806783f4844d782ffd0c901a1dafb (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260807 11:09:28.371989 25915 tablet_service.cc:1467] Processing CreateTablet for tablet 93d531084c9447b78cffb286d6b966b7 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260807 11:09:28.375653 25915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d531084c9447b78cffb286d6b966b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.252369 25901 tablet_service.cc:1467] Processing CreateTablet for tablet 986fe968b7be4eab9c480ec85c0384b0 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260807 11:09:28.376965 25901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 986fe968b7be4eab9c480ec85c0384b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.377343 26018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66d10b6b12404f0488b9f83c8080069f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.379467 26015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3d806783f4844d782ffd0c901a1dafb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.386565 25775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff0e05310b145358043bba0a9a8ba11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.393445 25906 tablet_service.cc:1467] Processing CreateTablet for tablet e702bb04fc98457f9d8ee704fcaeafcf (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260807 11:09:28.394727 25906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e702bb04fc98457f9d8ee704fcaeafcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.396303 25780 tablet_service.cc:1467] Processing CreateTablet for tablet 85d06c4d618c4aa6af896f216680dd68 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260807 11:09:28.397606 25780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85d06c4d618c4aa6af896f216680dd68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.284277 26014 tablet_service.cc:1467] Processing CreateTablet for tablet 99e5feeda1eb4affb2532bda1b0055cd (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260807 11:09:28.400388 25785 tablet_service.cc:1467] Processing CreateTablet for tablet 986fe968b7be4eab9c480ec85c0384b0 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260807 11:09:28.400576 25778 tablet_service.cc:1467] Processing CreateTablet for tablet 54a291aa09434b06adbb86586dacec0e (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260807 11:09:28.401623 25785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 986fe968b7be4eab9c480ec85c0384b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.402046 25909 tablet_service.cc:1467] Processing CreateTablet for tablet f5e378b8a19447d69c6c9433f071086e (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260807 11:09:28.402688 26011 tablet_service.cc:1467] Processing CreateTablet for tablet 6ecbb49904f34f5aba0936699ba75cfb (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260807 11:09:28.403213 25909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e378b8a19447d69c6c9433f071086e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.405102 25778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54a291aa09434b06adbb86586dacec0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.415215 25774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8330847089e41b0811e28dc30529234. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.417034 26032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c69ebea3aeb47d48ab8e083fdf71f7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.421079 26011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ecbb49904f34f5aba0936699ba75cfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.421612 26110 tablet_bootstrap.cc:492] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:28.422070 25783 tablet_service.cc:1467] Processing CreateTablet for tablet 76dbf7480b434c0ba6a902c817cf5455 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260807 11:09:28.423339 25783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76dbf7480b434c0ba6a902c817cf5455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.426563 26014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e5feeda1eb4affb2532bda1b0055cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.431072 26108 tablet_bootstrap.cc:654] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:28.433768 25907 tablet_service.cc:1467] Processing CreateTablet for tablet 99e5feeda1eb4affb2532bda1b0055cd (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260807 11:09:28.434587 26018 tablet_service.cc:1467] Processing CreateTablet for tablet adab2949574e4c6badfcf68e51b9c072 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260807 11:09:28.435827 26018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adab2949574e4c6badfcf68e51b9c072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.438731 25904 tablet_service.cc:1467] Processing CreateTablet for tablet b7c44ad3ac214b2d83c41bc5aa01f349 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260807 11:09:28.417092 26109 tablet_bootstrap.cc:654] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:28.434983 25907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e5feeda1eb4affb2532bda1b0055cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.454033 25779 tablet_service.cc:1467] Processing CreateTablet for tablet 3395e44a980a4ac2bc07304ae2525b35 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260807 11:09:28.455328 25779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3395e44a980a4ac2bc07304ae2525b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.459959 26024 tablet_service.cc:1467] Processing CreateTablet for tablet cccdf448d32448deae255de6bf7b4125 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260807 11:09:28.440693 25910 tablet_service.cc:1467] Processing CreateTablet for tablet a09a74fdae5d44c8a9b375e1cfdea1e9 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260807 11:09:28.468372 25910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a09a74fdae5d44c8a9b375e1cfdea1e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.470152 25903 tablet_service.cc:1467] Processing CreateTablet for tablet a805d50b18af4d5eb99f7c96e790e1dd (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260807 11:09:28.471400 25903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a805d50b18af4d5eb99f7c96e790e1dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.474143 25915 tablet_service.cc:1467] Processing CreateTablet for tablet 3c69ebea3aeb47d48ab8e083fdf71f7c (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260807 11:09:28.474874 25777 tablet_service.cc:1467] Processing CreateTablet for tablet 93d531084c9447b78cffb286d6b966b7 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260807 11:09:28.479947 25776 tablet_service.cc:1467] Processing CreateTablet for tablet f5e378b8a19447d69c6c9433f071086e (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260807 11:09:28.478092 25775 tablet_service.cc:1467] Processing CreateTablet for tablet e702bb04fc98457f9d8ee704fcaeafcf (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260807 11:09:28.428449 25913 tablet_service.cc:1467] Processing CreateTablet for tablet f35a17e1683e41f4b146284f5cf09c58 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260807 11:09:28.485435 25913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f35a17e1683e41f4b146284f5cf09c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.487929 25915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c69ebea3aeb47d48ab8e083fdf71f7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.489820 26110 tablet_bootstrap.cc:654] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:28.452277 26019 tablet_service.cc:1467] Processing CreateTablet for tablet 740366b034d646e09bc075abffba6286 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260807 11:09:28.497740 25904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7c44ad3ac214b2d83c41bc5aa01f349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.503607 26014 tablet_service.cc:1467] Processing CreateTablet for tablet 98fe28118d3d4770aa8de6f8d38ca61f (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260807 11:09:28.505820 25782 tablet_service.cc:1467] Processing CreateTablet for tablet f35a17e1683e41f4b146284f5cf09c58 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260807 11:09:28.507833 25774 tablet_service.cc:1467] Processing CreateTablet for tablet 99e5feeda1eb4affb2532bda1b0055cd (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260807 11:09:28.517628 26024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cccdf448d32448deae255de6bf7b4125. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.520143 25777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d531084c9447b78cffb286d6b966b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.534118 25780 tablet_service.cc:1467] Processing CreateTablet for tablet a09a74fdae5d44c8a9b375e1cfdea1e9 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260807 11:09:28.534981 26011 tablet_service.cc:1467] Processing CreateTablet for tablet 97a61636680c4b4a805a504dfcb4916f (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260807 11:09:28.527565 26025 tablet_service.cc:1467] Processing CreateTablet for tablet dacbdb7bffca4346a1e00e7b1e90fdea (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260807 11:09:28.539041 25774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e5feeda1eb4affb2532bda1b0055cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.499485 26015 tablet_service.cc:1467] Processing CreateTablet for tablet 674c5d3caeaf4ccabb23397e655b0e0a (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260807 11:09:28.545480 25851 heartbeater.cc:503] Master 127.23.120.190:37321 was elected leader, sending a full tablet report...
I20260807 11:09:28.545075 25776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e378b8a19447d69c6c9433f071086e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.553083 25775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e702bb04fc98457f9d8ee704fcaeafcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.555215 26026 tablet_service.cc:1467] Processing CreateTablet for tablet d8012276091e4b63b062cb7152208a16 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260807 11:09:28.553642 26018 tablet_service.cc:1467] Processing CreateTablet for tablet 01cf60138c9d41bbb8d914b0a2736151 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260807 11:09:28.558362 26109 tablet_bootstrap.cc:492] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:28.558820 26109 ts_tablet_manager.cc:1397] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.281s user 0.001s sys 0.018s
I20260807 11:09:28.562829 26019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 740366b034d646e09bc075abffba6286. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.567582 25782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f35a17e1683e41f4b146284f5cf09c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.571501 26109 raft_consensus.cc:348] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:28.572247 26109 raft_consensus.cc:374] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:28.581732 25780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a09a74fdae5d44c8a9b375e1cfdea1e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.572543 26109 raft_consensus.cc:729] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:28.583004 26109 consensus_queue.cc:260] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:28.586316 26108 tablet_bootstrap.cc:492] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:28.586771 26108 ts_tablet_manager.cc:1397] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.277s user 0.006s sys 0.013s
I20260807 11:09:28.589078 26015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674c5d3caeaf4ccabb23397e655b0e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.589699 25785 tablet_service.cc:1467] Processing CreateTablet for tablet 740366b034d646e09bc075abffba6286 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260807 11:09:28.589954 26032 tablet_service.cc:1467] Processing CreateTablet for tablet 437f2acbd46c43b2834bf00f08d22f65 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260807 11:09:28.590902 26014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98fe28118d3d4770aa8de6f8d38ca61f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.590969 25785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 740366b034d646e09bc075abffba6286. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.590579 25778 tablet_service.cc:1467] Processing CreateTablet for tablet b7c44ad3ac214b2d83c41bc5aa01f349 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260807 11:09:28.592016 25778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7c44ad3ac214b2d83c41bc5aa01f349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.592249 26018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01cf60138c9d41bbb8d914b0a2736151. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.593425 26011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97a61636680c4b4a805a504dfcb4916f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.594201 26032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 437f2acbd46c43b2834bf00f08d22f65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.595335 26025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dacbdb7bffca4346a1e00e7b1e90fdea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.605094 26110 tablet_bootstrap.cc:492] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:28.605729 26110 ts_tablet_manager.cc:1397] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.184s user 0.007s sys 0.019s
I20260807 11:09:28.559199 26016 tablet_service.cc:1467] Processing CreateTablet for tablet f31f65eddcdf48f6a64e0a1ddc0c6695 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260807 11:09:28.607440 26016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f31f65eddcdf48f6a64e0a1ddc0c6695. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.609750 26026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8012276091e4b63b062cb7152208a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.610111 25906 tablet_service.cc:1467] Processing CreateTablet for tablet e0766ccf18b84e41ac68eb6599a834c5 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260807 11:09:28.611850 25906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0766ccf18b84e41ac68eb6599a834c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.589170 25779 tablet_service.cc:1467] Processing CreateTablet for tablet a805d50b18af4d5eb99f7c96e790e1dd (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260807 11:09:28.623608 25779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a805d50b18af4d5eb99f7c96e790e1dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.627020 25900 tablet_service.cc:1467] Processing CreateTablet for tablet 51fb619c2ada475f817fbcb9739a401a (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260807 11:09:28.645175 26084 heartbeater.cc:503] Master 127.23.120.190:37321 was elected leader, sending a full tablet report...
I20260807 11:09:28.645157 26108 raft_consensus.cc:348] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } }
I20260807 11:09:28.646157 26108 raft_consensus.cc:374] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:28.646559 26108 raft_consensus.cc:729] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:28.647212 26108 consensus_queue.cc:260] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } }
I20260807 11:09:28.647737 25965 heartbeater.cc:503] Master 127.23.120.190:37321 was elected leader, sending a full tablet report...
I20260807 11:09:28.648545 25907 tablet_service.cc:1467] Processing CreateTablet for tablet 955db4a12dc04bfe8124047fc46823d3 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260807 11:09:28.628659 25900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51fb619c2ada475f817fbcb9739a401a. 1 dirs total, 0 dirs full, 0 dirs failed
W20260807 11:09:28.664958 25637 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 68890.1us
I20260807 11:09:28.646183 26109 ts_tablet_manager.cc:1428] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.086s user 0.004s sys 0.005s
I20260807 11:09:28.681685 25783 tablet_service.cc:1467] Processing CreateTablet for tablet e0766ccf18b84e41ac68eb6599a834c5 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260807 11:09:28.689590 25783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0766ccf18b84e41ac68eb6599a834c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.691507 25774 tablet_service.cc:1467] Processing CreateTablet for tablet 51fb619c2ada475f817fbcb9739a401a (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260807 11:09:28.693935 26108 ts_tablet_manager.cc:1428] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.107s user 0.007s sys 0.007s
I20260807 11:09:28.694953 26108 tablet_bootstrap.cc:492] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:28.697129 25907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 955db4a12dc04bfe8124047fc46823d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.706168 25775 tablet_service.cc:1467] Processing CreateTablet for tablet 3c69ebea3aeb47d48ab8e083fdf71f7c (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260807 11:09:28.707065 25901 tablet_service.cc:1467] Processing CreateTablet for tablet a3d806783f4844d782ffd0c901a1dafb (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260807 11:09:28.708308 25901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3d806783f4844d782ffd0c901a1dafb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.710706 25909 tablet_service.cc:1467] Processing CreateTablet for tablet 8467fdc1886745c1a7049de84b30bcc0 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260807 11:09:28.711949 25909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8467fdc1886745c1a7049de84b30bcc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.712241 25783 tablet_service.cc:1467] Processing CreateTablet for tablet 955db4a12dc04bfe8124047fc46823d3 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260807 11:09:28.713502 26109 tablet_bootstrap.cc:492] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:28.714219 26032 tablet_service.cc:1467] Processing CreateTablet for tablet 2f60eb5499d64c89b971928b13bcd1be (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260807 11:09:28.721669 25903 tablet_service.cc:1467] Processing CreateTablet for tablet 66d10b6b12404f0488b9f83c8080069f (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260807 11:09:28.722565 26032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f60eb5499d64c89b971928b13bcd1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.722347 25904 tablet_service.cc:1467] Processing CreateTablet for tablet 6ecbb49904f34f5aba0936699ba75cfb (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260807 11:09:28.723017 25903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66d10b6b12404f0488b9f83c8080069f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.723623 25904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ecbb49904f34f5aba0936699ba75cfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.727253 25915 tablet_service.cc:1467] Processing CreateTablet for tablet adab2949574e4c6badfcf68e51b9c072 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260807 11:09:28.728549 25915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adab2949574e4c6badfcf68e51b9c072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.732853 25774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51fb619c2ada475f817fbcb9739a401a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.734251 26110 raft_consensus.cc:348] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:28.735016 26110 raft_consensus.cc:374] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:28.735320 26110 raft_consensus.cc:729] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:28.735446 25783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 955db4a12dc04bfe8124047fc46823d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.736037 26110 consensus_queue.cc:260] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:28.739919 25778 tablet_service.cc:1467] Processing CreateTablet for tablet f40bb008c149418989fdcd7c1e67d554 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260807 11:09:28.779256 26026 tablet_service.cc:1467] Processing CreateTablet for tablet 737af11139a541029b9473a8bfdb5980 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260807 11:09:28.743896 26016 tablet_service.cc:1467] Processing CreateTablet for tablet 38360ae0ac1547f9887cae7534bb515b (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260807 11:09:28.745090 25775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c69ebea3aeb47d48ab8e083fdf71f7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.749928 26032 tablet_service.cc:1467] Processing CreateTablet for tablet b1765495856a47f09ddbb303f76c635f (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260807 11:09:28.798151 26032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1765495856a47f09ddbb303f76c635f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.799561 26015 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc5b1dfff1a4556abdd562afbebd768 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260807 11:09:28.753340 25785 tablet_service.cc:1467] Processing CreateTablet for tablet a3d806783f4844d782ffd0c901a1dafb (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260807 11:09:28.760004 25783 tablet_service.cc:1467] Processing CreateTablet for tablet 8467fdc1886745c1a7049de84b30bcc0 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260807 11:09:28.762943 26109 tablet_bootstrap.cc:654] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
W20260807 11:09:28.807193 25637 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 46162.9us
I20260807 11:09:28.809330 25900 tablet_service.cc:1467] Processing CreateTablet for tablet 674c5d3caeaf4ccabb23397e655b0e0a (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260807 11:09:28.810799 25900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674c5d3caeaf4ccabb23397e655b0e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.769694 25906 tablet_service.cc:1467] Processing CreateTablet for tablet 740366b034d646e09bc075abffba6286 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260807 11:09:28.811811 25906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 740366b034d646e09bc075abffba6286. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.774080 25776 tablet_service.cc:1467] Processing CreateTablet for tablet 6ecbb49904f34f5aba0936699ba75cfb (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260807 11:09:28.818367 26110 ts_tablet_manager.cc:1428] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.212s user 0.008s sys 0.000s
I20260807 11:09:28.819646 26018 tablet_service.cc:1467] Processing CreateTablet for tablet 998ef34204b64d2481698cc29e055cc2 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260807 11:09:28.780125 25910 tablet_service.cc:1467] Processing CreateTablet for tablet cccdf448d32448deae255de6bf7b4125 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260807 11:09:28.688108 25913 tablet_service.cc:1467] Processing CreateTablet for tablet f40bb008c149418989fdcd7c1e67d554 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260807 11:09:28.823923 25913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f40bb008c149418989fdcd7c1e67d554. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.824625 26110 tablet_bootstrap.cc:492] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:28.751699 25780 tablet_service.cc:1467] Processing CreateTablet for tablet 66d10b6b12404f0488b9f83c8080069f (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260807 11:09:28.832734 25904 tablet_service.cc:1467] Processing CreateTablet for tablet 97a61636680c4b4a805a504dfcb4916f (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260807 11:09:28.833704 25779 tablet_service.cc:1467] Processing CreateTablet for tablet 674c5d3caeaf4ccabb23397e655b0e0a (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260807 11:09:28.834302 25904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97a61636680c4b4a805a504dfcb4916f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.801020 26014 tablet_service.cc:1467] Processing CreateTablet for tablet 7be451f12275472a9c2c53877d9d1a0a (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260807 11:09:28.805294 25778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f40bb008c149418989fdcd7c1e67d554. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.805294 26016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38360ae0ac1547f9887cae7534bb515b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.766443 26108 tablet_bootstrap.cc:654] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:28.841813 26109 tablet_bootstrap.cc:492] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:28.842314 26109 ts_tablet_manager.cc:1397] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.129s user 0.010s sys 0.002s
I20260807 11:09:28.792145 25907 tablet_service.cc:1467] Processing CreateTablet for tablet 98fe28118d3d4770aa8de6f8d38ca61f (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260807 11:09:28.822870 26119 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:28.846115 26119 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } }
I20260807 11:09:28.839816 25783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8467fdc1886745c1a7049de84b30bcc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.841097 26014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be451f12275472a9c2c53877d9d1a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.850445 26119 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195)
I20260807 11:09:28.878247 25785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3d806783f4844d782ffd0c901a1dafb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.850943 25777 tablet_service.cc:1467] Processing CreateTablet for tablet adab2949574e4c6badfcf68e51b9c072 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260807 11:09:28.880467 25779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674c5d3caeaf4ccabb23397e655b0e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.881148 25780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66d10b6b12404f0488b9f83c8080069f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.880625 26025 tablet_service.cc:1467] Processing CreateTablet for tablet 61cb005020884d87b17995a28b8effd6 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260807 11:09:28.882254 25910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cccdf448d32448deae255de6bf7b4125. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.885125 26026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 737af11139a541029b9473a8bfdb5980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.859706 26109 raft_consensus.cc:348] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:28.858923 26024 tablet_service.cc:1467] Processing CreateTablet for tablet eb34326013f64140ac84b4fdaff38542 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260807 11:09:28.892606 25776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ecbb49904f34f5aba0936699ba75cfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.901067 26120 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:28.904284 25903 tablet_service.cc:1467] Processing CreateTablet for tablet f31f65eddcdf48f6a64e0a1ddc0c6695 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260807 11:09:28.915357 26120 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:28.916970 25782 tablet_service.cc:1467] Processing CreateTablet for tablet 98fe28118d3d4770aa8de6f8d38ca61f (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260807 11:09:28.917965 26120 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:28.918234 25782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98fe28118d3d4770aa8de6f8d38ca61f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.901232 26019 tablet_service.cc:1467] Processing CreateTablet for tablet dee1fb79ff3944c08f589c435f2f09a9 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260807 11:09:28.903311 25774 tablet_service.cc:1467] Processing CreateTablet for tablet cccdf448d32448deae255de6bf7b4125 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260807 11:09:28.901073 25907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98fe28118d3d4770aa8de6f8d38ca61f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.905527 25909 tablet_service.cc:1467] Processing CreateTablet for tablet 01cf60138c9d41bbb8d914b0a2736151 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260807 11:09:28.905618 26015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc5b1dfff1a4556abdd562afbebd768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.909833 26109 raft_consensus.cc:374] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:28.933843 25777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adab2949574e4c6badfcf68e51b9c072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.939368 26032 tablet_service.cc:1467] Processing CreateTablet for tablet bb5adbe0bf4741eb8b116a9bcf1cfc15 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260807 11:09:28.941289 25901 tablet_service.cc:1467] Processing CreateTablet for tablet dacbdb7bffca4346a1e00e7b1e90fdea (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260807 11:09:28.942133 26024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb34326013f64140ac84b4fdaff38542. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.933874 26109 raft_consensus.cc:729] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:28.955731 25903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f31f65eddcdf48f6a64e0a1ddc0c6695. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.971278 26109 consensus_queue.cc:260] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:28.974376 26109 ts_tablet_manager.cc:1428] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.132s user 0.007s sys 0.002s
I20260807 11:09:28.976814 25775 tablet_service.cc:1467] Processing CreateTablet for tablet 97a61636680c4b4a805a504dfcb4916f (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260807 11:09:28.977321 25909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01cf60138c9d41bbb8d914b0a2736151. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.976873 25774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cccdf448d32448deae255de6bf7b4125. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.982430 26018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 998ef34204b64d2481698cc29e055cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.988145 26108 tablet_bootstrap.cc:492] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:28.993211 26108 ts_tablet_manager.cc:1397] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.298s user 0.020s sys 0.015s
I20260807 11:09:28.993310 26019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee1fb79ff3944c08f589c435f2f09a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.996054 26108 raft_consensus.cc:348] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:28.999138 26108 raft_consensus.cc:374] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:28.999501 26108 raft_consensus.cc:729] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:28.997082 25901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dacbdb7bffca4346a1e00e7b1e90fdea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.003674 25913 tablet_service.cc:1467] Processing CreateTablet for tablet d8012276091e4b63b062cb7152208a16 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260807 11:09:29.007253 26115 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:29.007068 25783 tablet_service.cc:1467] Processing CreateTablet for tablet dacbdb7bffca4346a1e00e7b1e90fdea (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260807 11:09:29.007689 26025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61cb005020884d87b17995a28b8effd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.007687 26115 raft_consensus.cc:504] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.008390 25783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dacbdb7bffca4346a1e00e7b1e90fdea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:28.990810 26011 tablet_service.cc:1467] Processing CreateTablet for tablet d47975fbb6e3412aaf146ed0794f986c (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260807 11:09:29.022719 26011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d47975fbb6e3412aaf146ed0794f986c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.016649 26115 leader_election.cc:288] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:29.034204 25906 tablet_service.cc:1467] Processing CreateTablet for tablet 2f60eb5499d64c89b971928b13bcd1be (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260807 11:09:29.039369 26110 tablet_bootstrap.cc:654] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:29.041163 26109 tablet_bootstrap.cc:492] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:29.041890 25775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97a61636680c4b4a805a504dfcb4916f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.047156 26032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb5adbe0bf4741eb8b116a9bcf1cfc15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.050611 25785 tablet_service.cc:1467] Processing CreateTablet for tablet f31f65eddcdf48f6a64e0a1ddc0c6695 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260807 11:09:29.046829 25779 tablet_service.cc:1467] Processing CreateTablet for tablet 01cf60138c9d41bbb8d914b0a2736151 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260807 11:09:29.052234 25785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f31f65eddcdf48f6a64e0a1ddc0c6695. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.055814 25779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01cf60138c9d41bbb8d914b0a2736151. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.059648 25913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8012276091e4b63b062cb7152208a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.062903 25780 tablet_service.cc:1467] Processing CreateTablet for tablet d8012276091e4b63b062cb7152208a16 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260807 11:09:29.064199 25904 tablet_service.cc:1467] Processing CreateTablet for tablet 437f2acbd46c43b2834bf00f08d22f65 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260807 11:09:29.073503 26016 tablet_service.cc:1467] Processing CreateTablet for tablet 6eed5249afac4e16a4b17938ad6e1353 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260807 11:09:29.000419 26108 consensus_queue.cc:260] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.073959 25901 tablet_service.cc:1467] Processing CreateTablet for tablet 38360ae0ac1547f9887cae7534bb515b (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260807 11:09:29.075596 25907 tablet_service.cc:1467] Processing CreateTablet for tablet b1765495856a47f09ddbb303f76c635f (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260807 11:09:29.075896 26024 tablet_service.cc:1467] Processing CreateTablet for tablet e4de8b347f214a69bbac09d43ad38db7 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260807 11:09:29.078634 25777 tablet_service.cc:1467] Processing CreateTablet for tablet 437f2acbd46c43b2834bf00f08d22f65 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260807 11:09:29.092664 26024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4de8b347f214a69bbac09d43ad38db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.102526 26115 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:29.105881 26115 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.107812 26115 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:29.106436 25776 tablet_service.cc:1467] Processing CreateTablet for tablet 2f60eb5499d64c89b971928b13bcd1be (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260807 11:09:29.109724 26108 ts_tablet_manager.cc:1428] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.116s user 0.003s sys 0.005s
I20260807 11:09:29.110436 25776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f60eb5499d64c89b971928b13bcd1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.110709 26108 tablet_bootstrap.cc:492] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:29.111590 26016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eed5249afac4e16a4b17938ad6e1353. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.112438 25777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 437f2acbd46c43b2834bf00f08d22f65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.116685 26109 tablet_bootstrap.cc:654] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:29.122623 26015 tablet_service.cc:1467] Processing CreateTablet for tablet d5d5c9aff2844213a67d710585224083 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260807 11:09:29.130400 26015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5d5c9aff2844213a67d710585224083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.100562 25906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f60eb5499d64c89b971928b13bcd1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.134011 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:29.135088 25907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1765495856a47f09ddbb303f76c635f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.153770 26110 tablet_bootstrap.cc:492] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:29.154546 26110 ts_tablet_manager.cc:1397] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.330s user 0.025s sys 0.003s
W20260807 11:09:29.171800 25872 leader_election.cc:341] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:29.173326 25901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38360ae0ac1547f9887cae7534bb515b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.177901 25780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8012276091e4b63b062cb7152208a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.163932 25915 tablet_service.cc:1467] Processing CreateTablet for tablet 737af11139a541029b9473a8bfdb5980 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260807 11:09:29.199548 25904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 437f2acbd46c43b2834bf00f08d22f65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.200441 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:29.201432 25915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 737af11139a541029b9473a8bfdb5980. 1 dirs total, 0 dirs full, 0 dirs failed
W20260807 11:09:29.205751 25752 leader_election.cc:341] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:29.211241 25900 tablet_service.cc:1467] Processing CreateTablet for tablet 7be451f12275472a9c2c53877d9d1a0a (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260807 11:09:29.216329 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:09:29.218384 25869 leader_election.cc:341] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:29.218824 25869 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:29.219966 26119 raft_consensus.cc:2726] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:29.212644 25900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be451f12275472a9c2c53877d9d1a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.234843 26108 tablet_bootstrap.cc:654] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:29.243139 25910 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc5b1dfff1a4556abdd562afbebd768 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260807 11:09:29.249598 25910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc5b1dfff1a4556abdd562afbebd768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.251508 25909 tablet_service.cc:1467] Processing CreateTablet for tablet 998ef34204b64d2481698cc29e055cc2 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260807 11:09:29.245397 26032 tablet_service.cc:1467] Processing CreateTablet for tablet f9465d2caa3e43c8891f52350177df8d (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260807 11:09:29.263298 26032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9465d2caa3e43c8891f52350177df8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.270161 25779 tablet_service.cc:1467] Processing CreateTablet for tablet 38360ae0ac1547f9887cae7534bb515b (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260807 11:09:29.274926 25779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38360ae0ac1547f9887cae7534bb515b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.272855 25783 tablet_service.cc:1467] Processing CreateTablet for tablet b1765495856a47f09ddbb303f76c635f (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260807 11:09:29.281641 25783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1765495856a47f09ddbb303f76c635f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.283347 25777 tablet_service.cc:1467] Processing CreateTablet for tablet 737af11139a541029b9473a8bfdb5980 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260807 11:09:29.284583 25777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 737af11139a541029b9473a8bfdb5980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.285260 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:29.289646 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:29.301770 25909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 998ef34204b64d2481698cc29e055cc2. 1 dirs total, 0 dirs full, 0 dirs failed
W20260807 11:09:29.302668 25750 leader_election.cc:341] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:29.303094 25750 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:29.304189 26120 raft_consensus.cc:2726] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:29.289861 25901 tablet_service.cc:1467] Processing CreateTablet for tablet dee1fb79ff3944c08f589c435f2f09a9 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260807 11:09:29.290316 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:09:29.306330 25981 leader_election.cc:341] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:29.307685 25901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee1fb79ff3944c08f589c435f2f09a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.310559 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:29.310719 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:29.290609 25778 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc5b1dfff1a4556abdd562afbebd768 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260807 11:09:29.287007 25903 tablet_service.cc:1467] Processing CreateTablet for tablet eb34326013f64140ac84b4fdaff38542 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260807 11:09:29.293421 25776 tablet_service.cc:1467] Processing CreateTablet for tablet eb34326013f64140ac84b4fdaff38542 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260807 11:09:29.296212 26119 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:29.293670 25782 tablet_service.cc:1467] Processing CreateTablet for tablet 998ef34204b64d2481698cc29e055cc2 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260807 11:09:29.297120 26110 raft_consensus.cc:348] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.288276 25775 tablet_service.cc:1467] Processing CreateTablet for tablet 7be451f12275472a9c2c53877d9d1a0a (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260807 11:09:29.313211 26110 raft_consensus.cc:374] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:29.316498 26110 raft_consensus.cc:729] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:29.317291 26110 consensus_queue.cc:260] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.318435 25808 raft_consensus.cc:2454] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 0.
I20260807 11:09:29.319485 26110 ts_tablet_manager.cc:1428] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.164s user 0.007s sys 0.000s
I20260807 11:09:29.313328 26119 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.322863 26119 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:29.324020 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:09:29.314278 25981 leader_election.cc:341] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
W20260807 11:09:29.324908 25981 leader_election.cc:341] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:29.325268 25981 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:09:29.325942 26115 raft_consensus.cc:2726] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:29.326467 25776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb34326013f64140ac84b4fdaff38542. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.326372 25981 leader_election.cc:302] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
W20260807 11:09:29.327067 25872 leader_election.cc:341] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:29.327440 26115 raft_consensus.cc:2781] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:29.327629 25903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb34326013f64140ac84b4fdaff38542. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.327805 26115 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:29.328197 26115 raft_consensus.cc:3037] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:29.330574 25913 tablet_service.cc:1467] Processing CreateTablet for tablet bb5adbe0bf4741eb8b116a9bcf1cfc15 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260807 11:09:29.331924 25913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb5adbe0bf4741eb8b116a9bcf1cfc15. 1 dirs total, 0 dirs full, 0 dirs failed
W20260807 11:09:29.333374 25723 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 11:09:29.340917 26108 tablet_bootstrap.cc:492] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:29.341406 26108 ts_tablet_manager.cc:1397] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.231s user 0.013s sys 0.001s
I20260807 11:09:29.341255 25907 tablet_service.cc:1467] Processing CreateTablet for tablet d5d5c9aff2844213a67d710585224083 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260807 11:09:29.352994 25907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5d5c9aff2844213a67d710585224083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.353252 25775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be451f12275472a9c2c53877d9d1a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.359144 26109 tablet_bootstrap.cc:492] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:29.359573 26109 ts_tablet_manager.cc:1397] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.319s user 0.019s sys 0.002s
I20260807 11:09:29.360034 25774 tablet_service.cc:1467] Processing CreateTablet for tablet 61cb005020884d87b17995a28b8effd6 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260807 11:09:29.362231 26109 raft_consensus.cc:348] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.362438 26108 raft_consensus.cc:348] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.362986 26109 raft_consensus.cc:374] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:29.363193 26108 raft_consensus.cc:374] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:29.363320 26109 raft_consensus.cc:729] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:29.363524 26108 raft_consensus.cc:729] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:29.354139 26110 tablet_bootstrap.cc:492] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:29.364090 26109 consensus_queue.cc:260] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.365809 25778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc5b1dfff1a4556abdd562afbebd768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.385412 25780 tablet_service.cc:1467] Processing CreateTablet for tablet 6eed5249afac4e16a4b17938ad6e1353 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260807 11:09:29.369078 25777 tablet_service.cc:1467] Processing CreateTablet for tablet dee1fb79ff3944c08f589c435f2f09a9 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260807 11:09:29.358215 26115 raft_consensus.cc:504] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.394515 26115 leader_election.cc:288] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:29.396309 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:29.396945 25808 raft_consensus.cc:3037] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:29.374334 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:29.364238 26108 consensus_queue.cc:260] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:29.405051 25869 leader_election.cc:341] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:29.405521 25869 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:29.406569 26108 ts_tablet_manager.cc:1428] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.065s user 0.006s sys 0.000s
I20260807 11:09:29.406642 26119 raft_consensus.cc:2726] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:29.379521 25783 tablet_service.cc:1467] Processing CreateTablet for tablet bb5adbe0bf4741eb8b116a9bcf1cfc15 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260807 11:09:29.367172 25900 tablet_service.cc:1467] Processing CreateTablet for tablet 61cb005020884d87b17995a28b8effd6 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260807 11:09:29.389142 25774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61cb005020884d87b17995a28b8effd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.408073 25777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee1fb79ff3944c08f589c435f2f09a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.390928 26110 tablet_bootstrap.cc:654] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:29.408797 25783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb5adbe0bf4741eb8b116a9bcf1cfc15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.409765 26108 tablet_bootstrap.cc:492] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:29.414428 25909 tablet_service.cc:1467] Processing CreateTablet for tablet e4de8b347f214a69bbac09d43ad38db7 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260807 11:09:29.407459 25906 tablet_service.cc:1467] Processing CreateTablet for tablet 6eed5249afac4e16a4b17938ad6e1353 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260807 11:09:29.417819 25906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eed5249afac4e16a4b17938ad6e1353. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.420437 26119 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:29.425448 25900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61cb005020884d87b17995a28b8effd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.420704 25903 tablet_service.cc:1467] Processing CreateTablet for tablet f9465d2caa3e43c8891f52350177df8d (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260807 11:09:29.426524 25903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9465d2caa3e43c8891f52350177df8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.391081 25910 tablet_service.cc:1467] Processing CreateTablet for tablet d47975fbb6e3412aaf146ed0794f986c (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260807 11:09:29.404536 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:29.427043 26119 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:29.429096 25981 leader_election.cc:341] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:29.429744 26119 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:29.432121 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:29.433604 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:09:29.434870 25869 leader_election.cc:341] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
W20260807 11:09:29.435738 25872 leader_election.cc:341] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:29.436143 25872 leader_election.cc:302] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:29.437382 26119 raft_consensus.cc:2726] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:29.416165 25785 tablet_service.cc:1467] Processing CreateTablet for tablet d5d5c9aff2844213a67d710585224083 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260807 11:09:29.439723 25909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4de8b347f214a69bbac09d43ad38db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.424494 25808 raft_consensus.cc:2454] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:09:29.441879 25981 leader_election.cc:302] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:29.443002 26115 raft_consensus.cc:2781] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:29.443553 26115 raft_consensus.cc:686] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:09:29.445359 25910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d47975fbb6e3412aaf146ed0794f986c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.439954 26108 tablet_bootstrap.cc:654] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:29.442937 25782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 998ef34204b64d2481698cc29e055cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.447065 26115 consensus_queue.cc:237] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.465191 25785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5d5c9aff2844213a67d710585224083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.473542 25779 tablet_service.cc:1467] Processing CreateTablet for tablet d47975fbb6e3412aaf146ed0794f986c (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260807 11:09:29.478396 26109 ts_tablet_manager.cc:1428] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.119s user 0.021s sys 0.021s
I20260807 11:09:29.478058 25783 tablet_service.cc:1467] Processing CreateTablet for tablet e4de8b347f214a69bbac09d43ad38db7 (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260807 11:09:29.479753 26109 tablet_bootstrap.cc:492] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:29.482030 25775 tablet_service.cc:1467] Processing CreateTablet for tablet f9465d2caa3e43c8891f52350177df8d (DEFAULT_TABLE table=test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260807 11:09:29.487555 26109 tablet_bootstrap.cc:654] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:29.496757 25780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eed5249afac4e16a4b17938ad6e1353. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.502251 26110 tablet_bootstrap.cc:492] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:29.502835 26110 ts_tablet_manager.cc:1397] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.149s user 0.022s sys 0.018s
W20260807 11:09:29.503553 26087 tablet.cc:2369] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:09:29.508833 25655 catalog_manager.cc:5495] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260807 11:09:29.508144 26110 raft_consensus.cc:348] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } }
I20260807 11:09:29.511032 26110 raft_consensus.cc:374] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:29.511350 26110 raft_consensus.cc:729] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:29.512174 26110 consensus_queue.cc:260] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } }
I20260807 11:09:29.520218 25779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d47975fbb6e3412aaf146ed0794f986c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.521911 25783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4de8b347f214a69bbac09d43ad38db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.524986 25775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9465d2caa3e43c8891f52350177df8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:29.526283 26108 tablet_bootstrap.cc:492] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:29.536902 26108 ts_tablet_manager.cc:1397] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.127s user 0.007s sys 0.010s
I20260807 11:09:29.539887 26108 raft_consensus.cc:348] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.551374 26108 raft_consensus.cc:374] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:29.561511 26108 raft_consensus.cc:729] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:29.564412 26109 tablet_bootstrap.cc:492] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:29.563910 26108 consensus_queue.cc:260] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.562793 26110 ts_tablet_manager.cc:1428] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.059s user 0.008s sys 0.000s
I20260807 11:09:29.565403 26109 ts_tablet_manager.cc:1397] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.086s user 0.009s sys 0.009s
I20260807 11:09:29.572309 26110 tablet_bootstrap.cc:492] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:29.575610 26108 ts_tablet_manager.cc:1428] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.038s user 0.000s sys 0.007s
I20260807 11:09:29.571527 26109 raft_consensus.cc:348] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.581981 26108 tablet_bootstrap.cc:492] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:29.581915 26109 raft_consensus.cc:374] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:29.582726 26109 raft_consensus.cc:729] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:29.584182 26109 consensus_queue.cc:260] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.589341 26109 ts_tablet_manager.cc:1428] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.023s user 0.008s sys 0.005s
I20260807 11:09:29.590651 26109 tablet_bootstrap.cc:492] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:29.599905 26110 tablet_bootstrap.cc:654] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:29.603454 26109 tablet_bootstrap.cc:654] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:29.623620 26109 tablet_bootstrap.cc:492] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:29.624382 26109 ts_tablet_manager.cc:1397] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.017s
I20260807 11:09:29.628186 26108 tablet_bootstrap.cc:654] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:29.628410 26109 raft_consensus.cc:348] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.629647 26109 raft_consensus.cc:374] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:29.629980 26109 raft_consensus.cc:729] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:29.630807 26109 consensus_queue.cc:260] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.636816 26109 ts_tablet_manager.cc:1428] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.012s user 0.009s sys 0.001s
I20260807 11:09:29.638847 26109 tablet_bootstrap.cc:492] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:29.641775 26108 tablet_bootstrap.cc:492] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:29.642426 26108 ts_tablet_manager.cc:1397] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.061s user 0.012s sys 0.009s
I20260807 11:09:29.643133 26110 tablet_bootstrap.cc:492] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:29.644104 26110 ts_tablet_manager.cc:1397] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.072s user 0.024s sys 0.026s
I20260807 11:09:29.649298 26109 tablet_bootstrap.cc:654] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:29.649008 26108 raft_consensus.cc:348] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.650311 26108 raft_consensus.cc:374] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:29.650986 26108 raft_consensus.cc:729] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:29.650710 26110 raft_consensus.cc:348] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.653096 26110 raft_consensus.cc:374] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:29.653723 26110 raft_consensus.cc:729] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:29.652933 26108 consensus_queue.cc:260] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.654783 26110 consensus_queue.cc:260] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.670863 26108 ts_tablet_manager.cc:1428] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.028s user 0.015s sys 0.000s
I20260807 11:09:29.671834 26110 ts_tablet_manager.cc:1428] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.027s user 0.008s sys 0.007s
I20260807 11:09:29.672942 26108 tablet_bootstrap.cc:492] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:29.673676 26110 tablet_bootstrap.cc:492] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:29.688431 26110 tablet_bootstrap.cc:654] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:29.700575 26109 tablet_bootstrap.cc:492] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:29.701337 26109 ts_tablet_manager.cc:1397] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.063s user 0.026s sys 0.029s
I20260807 11:09:29.701742 26108 tablet_bootstrap.cc:654] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:29.705044 26109 raft_consensus.cc:348] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.706470 26109 raft_consensus.cc:374] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:29.707048 26110 tablet_bootstrap.cc:492] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:29.707126 26109 raft_consensus.cc:729] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:29.707670 26110 ts_tablet_manager.cc:1397] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.024s
I20260807 11:09:29.708508 26109 consensus_queue.cc:260] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.711920 26109 ts_tablet_manager.cc:1428] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260807 11:09:29.713513 26109 tablet_bootstrap.cc:492] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:29.714459 26108 tablet_bootstrap.cc:492] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:29.714937 26108 ts_tablet_manager.cc:1397] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.042s user 0.012s sys 0.012s
I20260807 11:09:29.716133 26115 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:29.711601 26110 raft_consensus.cc:348] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.716980 26110 raft_consensus.cc:374] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:29.716578 26115 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.717357 26110 raft_consensus.cc:729] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:29.717689 26108 raft_consensus.cc:348] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.718474 26108 raft_consensus.cc:374] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:29.718765 26108 raft_consensus.cc:729] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:29.718880 26115 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:29.718247 26110 consensus_queue.cc:260] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.722397 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:29.723363 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:29.723829 26110 ts_tablet_manager.cc:1428] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.016s user 0.005s sys 0.005s
W20260807 11:09:29.724084 25981 leader_election.cc:341] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:29.720753 26108 consensus_queue.cc:260] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:29.725486 25981 leader_election.cc:341] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:29.731732 26109 tablet_bootstrap.cc:654] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:29.733220 26110 tablet_bootstrap.cc:492] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:29.735080 26108 ts_tablet_manager.cc:1428] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20260807 11:09:29.735920 26115 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:29.736706 26108 tablet_bootstrap.cc:492] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:29.736788 26115 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.726328 25981 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:09:29.742980 26147 raft_consensus.cc:2726] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:29.740455 26115 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:29.744428 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:29.744724 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:29.746593 26110 tablet_bootstrap.cc:654] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
W20260807 11:09:29.761771 25981 leader_election.cc:341] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:29.765074 26108 tablet_bootstrap.cc:654] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
W20260807 11:09:29.765698 25981 leader_election.cc:341] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:29.766465 25981 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:09:29.764292 26109 tablet_bootstrap.cc:492] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:29.767573 26110 tablet_bootstrap.cc:492] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:29.767997 26109 ts_tablet_manager.cc:1397] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.055s user 0.007s sys 0.017s
I20260807 11:09:29.768179 26110 ts_tablet_manager.cc:1397] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.020s
I20260807 11:09:29.771833 26115 raft_consensus.cc:2726] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:29.773871 26110 raft_consensus.cc:348] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.775295 26110 raft_consensus.cc:374] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:29.775782 26110 raft_consensus.cc:729] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:29.777257 26110 consensus_queue.cc:260] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.778600 26109 raft_consensus.cc:348] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.779812 26109 raft_consensus.cc:374] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:29.780345 26109 raft_consensus.cc:729] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:29.782514 26110 ts_tablet_manager.cc:1428] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.014s user 0.007s sys 0.004s
I20260807 11:09:29.781844 26109 consensus_queue.cc:260] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.784432 26110 tablet_bootstrap.cc:492] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:29.782377 26108 tablet_bootstrap.cc:492] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:29.789515 26108 ts_tablet_manager.cc:1397] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.053s user 0.012s sys 0.018s
I20260807 11:09:29.793816 26108 raft_consensus.cc:348] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.795346 26108 raft_consensus.cc:374] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:29.795928 26108 raft_consensus.cc:729] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:29.798367 26109 ts_tablet_manager.cc:1428] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.030s user 0.008s sys 0.004s
I20260807 11:09:29.798424 26110 tablet_bootstrap.cc:654] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:29.797497 26108 consensus_queue.cc:260] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.800065 26109 tablet_bootstrap.cc:492] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:29.814077 26108 ts_tablet_manager.cc:1428] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.024s user 0.009s sys 0.003s
I20260807 11:09:29.815062 26109 tablet_bootstrap.cc:654] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:29.817762 26110 tablet_bootstrap.cc:492] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:29.818583 26110 ts_tablet_manager.cc:1397] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.020s
I20260807 11:09:29.820474 26108 tablet_bootstrap.cc:492] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:29.824129 26110 raft_consensus.cc:348] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.825467 26110 raft_consensus.cc:374] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:29.825922 26110 raft_consensus.cc:729] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:29.827523 26110 consensus_queue.cc:260] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.834728 26119 raft_consensus.cc:482] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:29.835427 26119 raft_consensus.cc:504] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:29.841002 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:29.842259 25808 raft_consensus.cc:2454] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
I20260807 11:09:29.843616 26110 ts_tablet_manager.cc:1428] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.024s user 0.011s sys 0.013s
I20260807 11:09:29.845451 26110 tablet_bootstrap.cc:492] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:29.845052 25869 leader_election.cc:302] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:09:29.840582 26108 tablet_bootstrap.cc:654] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:29.849249 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:29.850224 26046 raft_consensus.cc:2454] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
I20260807 11:09:29.852286 26119 leader_election.cc:288] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:29.853101 26119 raft_consensus.cc:2781] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:29.854126 26119 raft_consensus.cc:482] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:29.854957 26119 raft_consensus.cc:3037] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:29.859700 26110 tablet_bootstrap.cc:654] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:29.864530 26108 tablet_bootstrap.cc:492] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:29.865391 26108 ts_tablet_manager.cc:1397] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.045s user 0.013s sys 0.018s
I20260807 11:09:29.889652 26120 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:29.918687 26115 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:29.935635 26147 consensus_queue.cc:1035] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:30.031649 26120 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } }
I20260807 11:09:30.032876 26115 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.033675 26108 raft_consensus.cc:348] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.034559 26108 raft_consensus.cc:374] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:30.034857 26108 raft_consensus.cc:729] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:30.034986 26120 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:30.035271 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:30.035670 26120 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:30.035622 26108 consensus_queue.cc:260] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:30.037168 25752 leader_election.cc:341] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:30.036466 26120 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.084128 26115 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:30.114741 26108 ts_tablet_manager.cc:1428] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.243s user 0.028s sys 0.051s
I20260807 11:09:30.123188 26108 tablet_bootstrap.cc:492] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
W20260807 11:09:30.108309 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 263456us
I20260807 11:09:30.191437 26115 raft_consensus.cc:482] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:30.212945 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:30.218501 26109 tablet_bootstrap.cc:492] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:30.215993 26115 raft_consensus.cc:504] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.226706 26171 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:09:30.227464 25981 consensus_peers.cc:487] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 -> Peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Couldn't send request to peer 4fd6381544574ddca559ce63b9243efc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:30.227873 26110 tablet_bootstrap.cc:492] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:30.227298 26171 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.228348 26110 ts_tablet_manager.cc:1397] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.383s user 0.026s sys 0.042s
I20260807 11:09:30.223698 25931 raft_consensus.cc:2454] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 0.
I20260807 11:09:30.229975 26171 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:30.230508 26175 raft_consensus.cc:482] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:30.230751 26171 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:30.230841 26110 raft_consensus.cc:348] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.230959 26175 raft_consensus.cc:504] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.231675 26110 raft_consensus.cc:374] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:30.231974 26110 raft_consensus.cc:729] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:30.231463 26171 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.232599 26110 consensus_queue.cc:260] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.233379 26120 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:30.225652 26109 ts_tablet_manager.cc:1397] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.426s user 0.035s sys 0.035s
I20260807 11:09:30.233920 26171 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:30.234897 26110 ts_tablet_manager.cc:1428] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260807 11:09:30.235773 26110 tablet_bootstrap.cc:492] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:30.238523 26108 tablet_bootstrap.cc:654] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:30.242204 26110 tablet_bootstrap.cc:654] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:30.236445 26109 raft_consensus.cc:348] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.243552 26109 raft_consensus.cc:374] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:30.243820 26109 raft_consensus.cc:729] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:30.244431 26109 consensus_queue.cc:260] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.240163 26119 raft_consensus.cc:504] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.230031 25981 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:30.247516 26147 raft_consensus.cc:2781] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:30.247979 26147 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:30.248379 26147 raft_consensus.cc:3037] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:30.251405 26119 leader_election.cc:288] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:30.257320 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:30.257542 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:09:30.259855 25872 leader_election.cc:341] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:30.259897 26147 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:30.261277 25872 leader_election.cc:341] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:30.262684 26115 leader_election.cc:288] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:30.263953 26175 leader_election.cc:288] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:30.264370 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:30.270289 26109 ts_tablet_manager.cc:1428] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.036s user 0.006s sys 0.000s
I20260807 11:09:30.271020 26109 tablet_bootstrap.cc:492] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:30.272953 26147 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:30.260196 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:30.274947 26041 raft_consensus.cc:3037] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:30.265865 25931 raft_consensus.cc:2454] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 0.
I20260807 11:09:30.280467 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:30.284298 26046 raft_consensus.cc:2454] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
I20260807 11:09:30.285377 26041 raft_consensus.cc:2454] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:09:30.286005 25872 leader_election.cc:302] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:30.286243 25981 leader_election.cc:302] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:30.287149 25872 leader_election.cc:302] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:30.288287 26175 raft_consensus.cc:2781] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:30.289224 26175 raft_consensus.cc:482] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:30.289969 26175 raft_consensus.cc:3037] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:30.293268 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:30.294039 25931 raft_consensus.cc:3037] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:30.288429 26119 raft_consensus.cc:2781] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:30.296135 26119 raft_consensus.cc:686] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:09:30.297375 26109 tablet_bootstrap.cc:654] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:30.296983 26119 consensus_queue.cc:237] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.309254 26108 tablet_bootstrap.cc:492] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:30.309752 26108 ts_tablet_manager.cc:1397] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.187s user 0.018s sys 0.041s
I20260807 11:09:30.311733 26175 raft_consensus.cc:504] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.315662 26108 raft_consensus.cc:348] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.316407 26108 raft_consensus.cc:374] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:30.317328 26175 leader_election.cc:288] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:30.318284 26147 raft_consensus.cc:2781] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:30.323827 26147 raft_consensus.cc:482] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:30.324370 26147 raft_consensus.cc:3037] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:30.319267 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:30.316747 26108 raft_consensus.cc:729] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:30.320672 26110 tablet_bootstrap.cc:492] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:30.326598 26110 ts_tablet_manager.cc:1397] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.091s user 0.012s sys 0.001s
I20260807 11:09:30.326505 26041 raft_consensus.cc:3037] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:30.332031 26110 raft_consensus.cc:348] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.320211 25931 raft_consensus.cc:2454] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:09:30.332988 26110 raft_consensus.cc:374] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:30.333326 26110 raft_consensus.cc:729] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:30.334074 26110 consensus_queue.cc:260] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.334810 25981 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:30.336345 26115 raft_consensus.cc:2781] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:30.337975 26110 ts_tablet_manager.cc:1428] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.011s user 0.005s sys 0.003s
I20260807 11:09:30.339059 26041 raft_consensus.cc:2454] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:09:30.340823 26110 tablet_bootstrap.cc:492] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:30.341650 26147 raft_consensus.cc:504] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.353796 26109 tablet_bootstrap.cc:492] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:30.354367 26109 ts_tablet_manager.cc:1397] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.083s user 0.007s sys 0.014s
I20260807 11:09:30.343812 26108 consensus_queue.cc:260] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.343905 25872 leader_election.cc:302] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:30.352331 26110 tablet_bootstrap.cc:654] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:30.359333 26108 ts_tablet_manager.cc:1428] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.049s user 0.008s sys 0.002s
I20260807 11:09:30.370362 26108 tablet_bootstrap.cc:492] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:30.374366 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:30.380844 26108 tablet_bootstrap.cc:654] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:30.387346 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:30.388114 25798 raft_consensus.cc:2454] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
I20260807 11:09:30.388303 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:30.389271 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:09:30.389698 25869 leader_election.cc:341] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:30.390455 26041 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 0.
I20260807 11:09:30.390494 25869 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:30.392418 25752 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:30.400907 26120 raft_consensus.cc:2781] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:30.401521 26120 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:30.401680 26147 leader_election.cc:288] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:30.402001 26120 raft_consensus.cc:3037] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:30.394392 26180 raft_consensus.cc:2726] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:30.359750 26119 raft_consensus.cc:2781] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:30.399497 26109 raft_consensus.cc:348] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.405392 26109 raft_consensus.cc:374] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:30.359211 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:30.404227 26115 raft_consensus.cc:686] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:09:30.405618 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:30.405761 26109 raft_consensus.cc:729] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:30.408303 25801 raft_consensus.cc:3037] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:30.410652 26108 tablet_bootstrap.cc:492] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:30.411226 26108 ts_tablet_manager.cc:1397] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.041s user 0.015s sys 0.004s
I20260807 11:09:30.412326 26119 raft_consensus.cc:686] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:09:30.413074 26109 consensus_queue.cc:260] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.413894 26119 consensus_queue.cc:237] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.415771 26108 raft_consensus.cc:348] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.416694 26108 raft_consensus.cc:374] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:30.417073 26108 raft_consensus.cc:729] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:30.361250 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:30.417802 26108 consensus_queue.cc:260] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:30.419569 25981 leader_election.cc:341] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:30.407215 25931 raft_consensus.cc:3037] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:30.422405 26108 ts_tablet_manager.cc:1428] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.011s user 0.006s sys 0.003s
I20260807 11:09:30.424042 26109 ts_tablet_manager.cc:1428] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.069s user 0.006s sys 0.002s
I20260807 11:09:30.426221 26109 tablet_bootstrap.cc:492] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:30.427556 26108 tablet_bootstrap.cc:492] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:30.424611 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:30.430119 26174 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:30.424925 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:30.441469 26176 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:30.442348 26176 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.447810 26110 tablet_bootstrap.cc:492] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:30.448431 26110 ts_tablet_manager.cc:1397] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.108s user 0.017s sys 0.009s
I20260807 11:09:30.166376 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:30.407904 25869 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:09:30.450157 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:30.453279 25925 raft_consensus.cc:2454] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 0.
I20260807 11:09:30.454181 26109 tablet_bootstrap.cc:654] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:30.407178 26115 consensus_queue.cc:237] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.453344 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:30.445185 25932 raft_consensus.cc:2454] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 0.
I20260807 11:09:30.456020 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:09:30.459268 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 66557us
I20260807 11:09:30.460435 26180 raft_consensus.cc:2781] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260807 11:09:30.460883 25869 leader_election.cc:341] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:30.461081 26180 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 11:09:30.461166 25752 leader_election.cc:341] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:30.433413 26174 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.461704 26180 raft_consensus.cc:3037] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:30.451943 26147 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:09:30.458516 25750 leader_election.cc:341] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
W20260807 11:09:30.462410 25869 leader_election.cc:341] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
W20260807 11:09:31.128598 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 666174us
I20260807 11:09:30.452250 26176 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:30.477274 25801 raft_consensus.cc:2454] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:09:30.493290 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:31.131127 25797 raft_consensus.cc:3037] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:30.558252 26174 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:30.471611 25750 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:09:31.149420 26199 consensus_queue.cc:1035] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260807 11:09:31.149799 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 678336us
I20260807 11:09:30.557376 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:30.648043 26180 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.636097 26108 tablet_bootstrap.cc:654] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:30.641203 26110 raft_consensus.cc:348] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.618098 26120 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.465536 26147 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.734977 26171 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:31.185189 26171 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:30.977813 26195 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:31.105576 26109 tablet_bootstrap.cc:492] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:31.190032 26109 ts_tablet_manager.cc:1397] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.764s user 0.012s sys 0.029s
I20260807 11:09:31.189734 26195 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.190752 26120 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:31.105300 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:31.116297 26196 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:31.121224 25660 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:40714 (request call id 16) took 1263 ms. Trace:
I20260807 11:09:31.193809 25660 rpcz_store.cc:276] 0807 11:09:29.857890 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:09:29.858137 (+ 247us) service_pool.cc:227] Handling call
0807 11:09:31.121147 (+1263010us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6385,"reported_tablets":1}
I20260807 11:09:31.194427 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:09:31.133728 25931 consensus_meta.cc:287] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.712s user 0.016s sys 0.020s
I20260807 11:09:31.195580 26042 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
I20260807 11:09:31.201345 25931 raft_consensus.cc:2454] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:09:31.143126 26119 consensus_queue.cc:1035] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:31.154531 25797 raft_consensus.cc:2454] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:09:31.157121 26174 raft_consensus.cc:2781] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:31.204958 26174 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:31.205487 26174 raft_consensus.cc:3037] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:31.157882 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:09:31.160804 25752 leader_election.cc:341] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:31.161417 26180 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:31.162017 25750 leader_election.cc:302] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
W20260807 11:09:31.210445 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 49742.3us
W20260807 11:09:31.211436 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 49620.4us
I20260807 11:09:31.211397 26195 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:31.174787 25659 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:40716 (request call id 15) took 1344 ms. Trace:
I20260807 11:09:31.165092 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:31.177803 26110 raft_consensus.cc:374] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:31.203550 25872 leader_election.cc:302] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:09:31.166126 25723 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 11:09:31.213462 26120 raft_consensus.cc:2781] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:31.221725 26120 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:31.222388 26120 raft_consensus.cc:3037] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:31.218262 26110 raft_consensus.cc:729] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:31.218760 26109 raft_consensus.cc:348] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.195905 26196 raft_consensus.cc:504] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.219894 26147 leader_election.cc:288] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:31.219803 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:31.217725 25659 rpcz_store.cc:276] 0807 11:09:29.820943 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:09:29.821707 (+ 764us) service_pool.cc:227] Handling call
0807 11:09:31.165503 (+1343796us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":767500,"reported_tablets":1}
I20260807 11:09:31.223587 26109 raft_consensus.cc:374] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:31.223979 26110 consensus_queue.cc:260] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.233896 26109 raft_consensus.cc:729] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:31.235143 26180 raft_consensus.cc:2781] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:31.235816 26110 ts_tablet_manager.cc:1428] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.787s user 0.011s sys 0.002s
W20260807 11:09:31.236567 26110 ts_tablet_manager.cc:1460] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815: Tablet startup took 787ms
I20260807 11:09:31.235723 26180 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:31.239843 26180 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:31.236455 26039 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 0.
W20260807 11:09:31.243400 25869 consensus_peers.cc:487] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc -> Peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Couldn't send request to peer 606d49e053a346e486f48cb31832b815. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:31.243448 26196 leader_election.cc:288] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:09:31.244158 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 101112us
I20260807 11:09:31.244961 25752 leader_election.cc:302] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:31.244916 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:31.245054 25932 raft_consensus.cc:3037] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:31.246803 26147 consensus_queue.cc:1035] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:31.247938 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:31.197206 26171 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:31.237331 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:09:31.238108 26110 ts_tablet_manager.cc:1462] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815: Trace:
0807 11:09:28.361682 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:09:28.361822 (+ 140us) service_pool.cc:227] Handling call
0807 11:09:28.414493 (+ 52671us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0807 11:09:28.414780 (+ 287us) ts_tablet_manager.cc:676] Creating new metadata...
0807 11:09:28.435345 (+ 20565us) tablet_metadata.cc:641] Metadata flushed
0807 11:09:28.466801 (+ 31456us) tablet_replica.cc:205] Creating consensus instance
0807 11:09:28.506673 (+ 39872us) inbound_call.cc:173] Queueing success response
0807 11:09:30.340414 (+1833741us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0807 11:09:30.448844 (+108430us) ts_tablet_manager.cc:1429] Starting tablet replica
0807 11:09:30.638386 (+189542us) tablet_replica.cc:259] Starting instrumentation
0807 11:09:30.639307 (+ 921us) tablet_replica.cc:278] Starting consensus
Metrics: {"tablet-open.queue_time_us":1833814}
W20260807 11:09:31.234138 25750 leader_election.cc:341] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:31.251350 26109 consensus_queue.cc:260] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.252784 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:31.261839 26110 tablet_bootstrap.cc:492] T 54a291aa09434b06adbb86586dacec0e P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:31.264483 26109 ts_tablet_manager.cc:1428] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.074s user 0.007s sys 0.000s
I20260807 11:09:31.265239 26109 tablet_bootstrap.cc:492] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
W20260807 11:09:31.260986 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 91071us
I20260807 11:09:31.268316 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:31.269273 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:31.269452 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:31.265967 25750 leader_election.cc:302] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:31.273106 26176 raft_consensus.cc:2726] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260807 11:09:31.272316 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 45294.1us
I20260807 11:09:31.265913 26041 raft_consensus.cc:3037] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
W20260807 11:09:31.275395 25750 leader_election.cc:341] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
W20260807 11:09:31.278420 26200 log.cc:924] Time spent T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.065s user 0.000s sys 0.001s
I20260807 11:09:31.260217 26042 raft_consensus.cc:3037] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:31.269193 25801 raft_consensus.cc:3037] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:31.266165 26195 raft_consensus.cc:2781] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:31.312829 26171 consensus_queue.cc:1035] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260807 11:09:31.303992 26176 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:31.314911 26176 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.307317 26195 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:31.326814 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:31.335425 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:31.325229 26195 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:31.341986 26176 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:31.342823 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:31.315243 25981 leader_election.cc:302] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:09:31.344184 26180 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.355826 25926 raft_consensus.cc:2454] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 0.
I20260807 11:09:31.372886 26115 raft_consensus.cc:2781] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:31.384120 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:09:31.377415 25981 leader_election.cc:341] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:31.369289 26120 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.378886 26202 consensus_queue.cc:1035] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260807 11:09:31.371901 25869 leader_election.cc:302] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:31.398141 26180 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:31.386976 26115 raft_consensus.cc:686] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:09:31.407833 25931 raft_consensus.cc:2379] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 1: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:09:31.425396 25660 catalog_manager.cc:5495] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } }
I20260807 11:09:31.427997 26109 tablet_bootstrap.cc:654] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:31.427625 25660 catalog_manager.cc:5495] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } }
W20260807 11:09:31.408433 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 54090us
I20260807 11:09:31.410557 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:31.408608 26201 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:31.419847 25981 leader_election.cc:302] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:09:31.392730 25750 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:09:31.438414 26039 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
W20260807 11:09:31.445091 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 55856.5us
I20260807 11:09:31.449179 26120 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:31.459362 26108 tablet_bootstrap.cc:492] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:31.459945 26108 ts_tablet_manager.cc:1397] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 1.033s user 0.035s sys 0.014s
I20260807 11:09:31.443429 26201 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.458360 25660 catalog_manager.cc:5495] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
W20260807 11:09:31.437863 25752 leader_election.cc:341] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:31.469826 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:31.471199 26201 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:31.478821 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:31.479527 25931 raft_consensus.cc:3037] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
W20260807 11:09:31.465384 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 45697.7us
W20260807 11:09:31.486135 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 77227.9us
I20260807 11:09:31.490419 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:31.491941 26108 raft_consensus.cc:348] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.494261 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:31.498050 26108 raft_consensus.cc:374] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:31.502288 26108 raft_consensus.cc:729] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:31.503161 26108 consensus_queue.cc:260] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.511765 26208 raft_consensus.cc:2781] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:31.475989 25801 raft_consensus.cc:2454] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:09:31.482674 26042 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:09:31.512895 26208 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 11:09:31.531401 25981 leader_election.cc:341] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:31.560235 26108 ts_tablet_manager.cc:1428] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.100s user 0.008s sys 0.003s
I20260807 11:09:31.572302 25869 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:09:31.501713 26115 consensus_queue.cc:237] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.539040 26208 raft_consensus.cc:3037] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:31.581866 26195 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.597836 25803 raft_consensus.cc:2379] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 1: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:09:31.599087 26108 tablet_bootstrap.cc:492] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:31.583822 26171 raft_consensus.cc:2781] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:31.590106 26180 raft_consensus.cc:2726] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:31.602356 26039 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:09:31.597847 26110 tablet_bootstrap.cc:654] T 54a291aa09434b06adbb86586dacec0e P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:31.602878 25752 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:31.606352 26147 raft_consensus.cc:2726] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:31.607175 26171 raft_consensus.cc:686] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:09:31.610589 26171 consensus_queue.cc:237] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.607487 26120 raft_consensus.cc:2781] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:31.612623 26120 raft_consensus.cc:686] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:09:31.614348 26120 consensus_queue.cc:237] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.594388 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:09:31.605479 25981 leader_election.cc:341] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
W20260807 11:09:31.660140 25847 debug-util.cc:398] Leaking SignalData structure 0x7b0800119da0 after lost signal to thread 24038
I20260807 11:09:31.666749 26195 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:31.633301 25981 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:09:31.683496 26147 raft_consensus.cc:2726] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260807 11:09:31.683758 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 79932.5us
I20260807 11:09:31.683565 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:31.685328 25926 raft_consensus.cc:2379] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 1: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
W20260807 11:09:31.686467 25847 debug-util.cc:398] Leaking SignalData structure 0x7b08001ded80 after lost signal to thread 25705
W20260807 11:09:31.688061 25847 debug-util.cc:398] Leaking SignalData structure 0x7b080015ea00 after lost signal to thread 25850
I20260807 11:09:31.669798 26041 raft_consensus.cc:2454] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:09:31.688347 25931 raft_consensus.cc:2454] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:09:31.692402 25872 leader_election.cc:302] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:09:31.676615 26174 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } }
I20260807 11:09:31.695309 26224 raft_consensus.cc:2781] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:31.696197 26224 raft_consensus.cc:686] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
W20260807 11:09:31.698127 25752 leader_election.cc:341] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
W20260807 11:09:31.711277 25847 debug-util.cc:398] Leaking SignalData structure 0x7b08001f8aa0 after lost signal to thread 25964
W20260807 11:09:31.718935 25847 debug-util.cc:398] Leaking SignalData structure 0x7b080015b160 after lost signal to thread 26083
I20260807 11:09:31.726097 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:31.714699 25750 leader_election.cc:302] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:09:31.727404 26041 raft_consensus.cc:2379] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 1: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:09:31.728008 26195 raft_consensus.cc:2781] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:31.728797 26195 raft_consensus.cc:686] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:09:31.729045 25752 leader_election.cc:302] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:31.730360 26174 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:31.736711 26176 raft_consensus.cc:2726] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260807 11:09:31.730100 26195 consensus_queue.cc:237] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.739936 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:31.742105 25932 raft_consensus.cc:3037] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:31.749670 26109 tablet_bootstrap.cc:492] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:31.750296 26109 ts_tablet_manager.cc:1397] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.485s user 0.026s sys 0.122s
I20260807 11:09:31.756186 26119 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:31.765200 26119 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.774912 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:09:31.778425 25752 leader_election.cc:341] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:31.740186 26224 consensus_queue.cc:237] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.783509 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:31.789336 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:31.790616 26041 raft_consensus.cc:2454] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
I20260807 11:09:31.786635 26119 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:09:31.793164 25869 leader_election.cc:341] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:31.794122 25872 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:09:31.796794 26119 raft_consensus.cc:2781] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:31.797333 26119 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:31.797741 26119 raft_consensus.cc:3037] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:31.808888 26108 tablet_bootstrap.cc:654] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:31.816836 25932 raft_consensus.cc:2454] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:09:31.819763 26110 tablet_bootstrap.cc:492] T 54a291aa09434b06adbb86586dacec0e P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:31.811753 26208 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.824239 26110 ts_tablet_manager.cc:1397] T 54a291aa09434b06adbb86586dacec0e P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.563s user 0.025s sys 0.016s
I20260807 11:09:31.825985 25750 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:09:31.830730 26109 raft_consensus.cc:348] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.827647 26119 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.834738 26109 raft_consensus.cc:374] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:31.853288 26109 raft_consensus.cc:729] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:31.854344 26109 consensus_queue.cc:260] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.855468 26119 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:31.856819 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:31.845216 26195 raft_consensus.cc:2781] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:31.857892 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:31.858651 26195 raft_consensus.cc:686] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:09:31.858772 26041 raft_consensus.cc:3037] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:31.857323 26109 ts_tablet_manager.cc:1428] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.107s user 0.011s sys 0.000s
I20260807 11:09:31.861326 26109 tablet_bootstrap.cc:492] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:31.860445 26195 consensus_queue.cc:237] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } }
I20260807 11:09:31.862074 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:31.863389 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:31.864282 25932 raft_consensus.cc:3037] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:31.860122 26208 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:31.838290 26110 raft_consensus.cc:348] T 54a291aa09434b06adbb86586dacec0e P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.866516 26110 raft_consensus.cc:374] T 54a291aa09434b06adbb86586dacec0e P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:31.866864 26110 raft_consensus.cc:729] T 54a291aa09434b06adbb86586dacec0e P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:31.867551 26110 consensus_queue.cc:260] T 54a291aa09434b06adbb86586dacec0e P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:31.867905 25752 leader_election.cc:341] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
W20260807 11:09:31.867940 25869 leader_election.cc:341] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:31.870237 26110 ts_tablet_manager.cc:1428] T 54a291aa09434b06adbb86586dacec0e P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.037s user 0.007s sys 0.002s
I20260807 11:09:31.870978 26115 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:31.871995 26115 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.871455 26110 tablet_bootstrap.cc:492] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:31.877002 26115 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:31.879740 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:31.879837 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:09:31.882267 25981 leader_election.cc:341] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
W20260807 11:09:31.883405 25981 leader_election.cc:341] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:31.884081 25981 leader_election.cc:302] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:09:31.885455 26115 raft_consensus.cc:2726] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:31.904273 25932 raft_consensus.cc:2454] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:09:31.906704 25750 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:09:31.907773 26195 raft_consensus.cc:2781] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:31.909070 26195 raft_consensus.cc:686] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:09:31.910773 26109 tablet_bootstrap.cc:654] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:31.910444 26195 consensus_queue.cc:237] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.918542 26110 tablet_bootstrap.cc:654] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:31.926542 26041 raft_consensus.cc:2454] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:09:31.929535 25872 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:09:31.931684 26224 raft_consensus.cc:2781] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:31.932446 26224 raft_consensus.cc:686] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:09:31.937995 26224 consensus_queue.cc:237] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.954383 26108 tablet_bootstrap.cc:492] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:31.955190 26108 ts_tablet_manager.cc:1397] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.356s user 0.021s sys 0.030s
I20260807 11:09:31.980517 26108 raft_consensus.cc:348] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.985249 26108 raft_consensus.cc:374] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:31.985736 26108 raft_consensus.cc:729] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:31.986788 26108 consensus_queue.cc:260] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:31.989221 26108 ts_tablet_manager.cc:1428] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.034s user 0.017s sys 0.016s
I20260807 11:09:31.973347 25658 catalog_manager.cc:5495] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 reported cstate change: term changed from 0 to 1, leader changed from <none> to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:31.990360 26108 tablet_bootstrap.cc:492] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:31.991688 26109 tablet_bootstrap.cc:492] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:31.992174 26109 ts_tablet_manager.cc:1397] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.131s user 0.013s sys 0.016s
I20260807 11:09:31.997190 26109 raft_consensus.cc:348] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:32.005378 26109 raft_consensus.cc:374] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:32.005762 26109 raft_consensus.cc:729] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:32.006601 26109 consensus_queue.cc:260] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:32.013130 26109 ts_tablet_manager.cc:1428] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.021s user 0.007s sys 0.007s
I20260807 11:09:32.014245 26109 tablet_bootstrap.cc:492] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:32.020372 26108 tablet_bootstrap.cc:654] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:32.024987 25660 catalog_manager.cc:5495] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260807 11:09:32.033977 26110 tablet_bootstrap.cc:492] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:32.034480 26110 ts_tablet_manager.cc:1397] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.163s user 0.027s sys 0.033s
I20260807 11:09:32.037945 26110 raft_consensus.cc:348] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:32.038697 26110 raft_consensus.cc:374] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:32.038959 26110 raft_consensus.cc:729] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:32.039700 26110 consensus_queue.cc:260] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:32.042024 26110 ts_tablet_manager.cc:1428] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260807 11:09:32.042912 26110 tablet_bootstrap.cc:492] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:32.043996 26109 tablet_bootstrap.cc:654] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:32.054782 26110 tablet_bootstrap.cc:654] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:32.060348 26195 raft_consensus.cc:482] T 54a291aa09434b06adbb86586dacec0e P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:32.061157 26195 raft_consensus.cc:504] T 54a291aa09434b06adbb86586dacec0e P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:32.062772 25806 raft_consensus.cc:3037] T 54a291aa09434b06adbb86586dacec0e P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:32.066574 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:32.066677 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:32.067873 25932 raft_consensus.cc:2379] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 1: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:09:32.069372 25750 leader_election.cc:302] T 54a291aa09434b06adbb86586dacec0e P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:32.069666 26195 leader_election.cc:288] T 54a291aa09434b06adbb86586dacec0e P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:32.078382 26109 tablet_bootstrap.cc:492] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:32.078944 26109 ts_tablet_manager.cc:1397] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.065s user 0.008s sys 0.012s
I20260807 11:09:32.083590 26109 raft_consensus.cc:348] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:32.084579 26109 raft_consensus.cc:374] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:32.086229 26109 raft_consensus.cc:729] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:32.087045 26109 consensus_queue.cc:260] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:32.089788 26208 raft_consensus.cc:2726] T 54a291aa09434b06adbb86586dacec0e P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:32.090184 26110 tablet_bootstrap.cc:492] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:32.093232 26110 ts_tablet_manager.cc:1397] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.050s user 0.012s sys 0.011s
I20260807 11:09:32.093290 26108 tablet_bootstrap.cc:492] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:32.093794 26108 ts_tablet_manager.cc:1397] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.104s user 0.018s sys 0.022s
I20260807 11:09:32.098023 26115 consensus_queue.cc:1035] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:32.098214 26108 raft_consensus.cc:348] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:32.099159 26108 raft_consensus.cc:374] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:32.101676 26109 ts_tablet_manager.cc:1428] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.022s user 0.004s sys 0.004s
I20260807 11:09:32.100651 26108 raft_consensus.cc:729] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:32.103302 26108 consensus_queue.cc:260] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:32.104987 26195 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:32.105356 26110 raft_consensus.cc:348] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:32.106195 26110 raft_consensus.cc:374] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:32.105626 26195 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:32.106487 26110 raft_consensus.cc:729] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:32.107264 26110 consensus_queue.cc:260] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:32.109925 26110 ts_tablet_manager.cc:1428] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.015s user 0.003s sys 0.004s
I20260807 11:09:32.110008 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:32.110955 26110 tablet_bootstrap.cc:492] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:32.112053 26109 tablet_bootstrap.cc:492] T 9c6408e9acee46adacd2bbd81669f27a P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:32.117175 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:32.119529 26041 raft_consensus.cc:2379] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 1: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:09:32.118499 26195 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:32.121438 25752 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:32.123075 26108 ts_tablet_manager.cc:1428] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.029s user 0.008s sys 0.000s
I20260807 11:09:32.123345 26195 raft_consensus.cc:3037] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:32.124199 26108 tablet_bootstrap.cc:492] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:32.090130 25659 catalog_manager.cc:5495] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:32.130868 25659 catalog_manager.cc:5495] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:32.139416 25659 catalog_manager.cc:5495] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:32.153187 26208 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:32.154546 26208 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:32.157172 26108 tablet_bootstrap.cc:654] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:32.159011 26208 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:32.159386 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:09:32.161427 25750 leader_election.cc:341] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:32.162031 26195 raft_consensus.cc:2726] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:32.162433 26224 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:32.166244 26224 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:32.161317 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:32.177253 26041 raft_consensus.cc:2454] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 0.
I20260807 11:09:32.179149 25752 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:32.179980 26195 consensus_queue.cc:1035] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:32.180971 26208 raft_consensus.cc:2781] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:32.181497 26208 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:32.182164 26208 raft_consensus.cc:3037] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:32.184716 26109 tablet_bootstrap.cc:654] T 9c6408e9acee46adacd2bbd81669f27a P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:32.186026 26115 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:32.186782 26115 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:32.188805 26110 tablet_bootstrap.cc:654] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:32.193097 26115 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:32.196703 26224 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:32.197727 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:32.199074 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:32.202253 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:32.203647 25932 raft_consensus.cc:2379] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 1: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:09:32.209749 25660 catalog_manager.cc:5495] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 reported cstate change: term changed from 0 to 1, leader changed from <none> to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:32.211935 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:32.192479 26147 consensus_queue.cc:1035] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20260807 11:09:32.213450 25660 catalog_manager.cc:5495] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 reported cstate change: term changed from 0 to 1, leader changed from <none> to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:32.214133 26208 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:32.216084 25872 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:32.217532 26224 raft_consensus.cc:2726] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:32.220448 25926 raft_consensus.cc:3037] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:32.223764 26039 raft_consensus.cc:3037] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
W20260807 11:09:32.225392 25723 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 11:09:32.215755 25660 catalog_manager.cc:5495] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 reported cstate change: term changed from 0 to 1, leader changed from <none> to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } }
I20260807 11:09:32.228574 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:32.232493 26208 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:32.234102 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:32.236482 26042 raft_consensus.cc:3037] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:32.231029 26224 consensus_queue.cc:1035] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:32.275709 26147 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:32.276597 26147 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:32.284241 26147 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:32.291983 26119 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:32.292706 26119 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:32.240998 25981 leader_election.cc:302] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:09:32.296104 26171 consensus_queue.cc:1035] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
W20260807 11:09:32.301659 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 64500.2us
I20260807 11:09:32.303445 26110 tablet_bootstrap.cc:492] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:32.303870 26110 ts_tablet_manager.cc:1397] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.193s user 0.014s sys 0.008s
I20260807 11:09:32.310338 26119 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:32.311622 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:32.312587 25808 raft_consensus.cc:2454] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
W20260807 11:09:32.305197 25750 leader_election.cc:341] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:32.321188 25869 leader_election.cc:302] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:09:32.333362 26224 raft_consensus.cc:2781] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:32.356050 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:32.344134 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:32.343729 26195 consensus_queue.cc:1035] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260807 11:09:32.344429 26208 consensus_queue.cc:1035] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.033s
I20260807 11:09:32.370085 25808 raft_consensus.cc:2454] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 0.
I20260807 11:09:32.376746 25981 leader_election.cc:302] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:32.377542 26042 raft_consensus.cc:2454] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:09:32.378118 26109 tablet_bootstrap.cc:492] T 9c6408e9acee46adacd2bbd81669f27a P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:32.378660 26109 ts_tablet_manager.cc:1397] T 9c6408e9acee46adacd2bbd81669f27a P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.267s user 0.023s sys 0.086s
I20260807 11:09:32.358373 26224 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:32.372478 26110 raft_consensus.cc:348] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:32.381225 26110 raft_consensus.cc:374] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:32.381495 26110 raft_consensus.cc:729] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:32.378137 26198 raft_consensus.cc:2781] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:32.382995 26110 consensus_queue.cc:260] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:32.380235 26224 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:32.402606 26198 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:32.404354 26198 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:32.385052 26147 raft_consensus.cc:2726] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:32.392159 25752 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:32.409623 26195 raft_consensus.cc:2781] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:32.410516 26195 raft_consensus.cc:686] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:09:32.419281 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:32.422577 26237 consensus_queue.cc:1035] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:32.426819 26208 consensus_queue.cc:1035] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:32.427343 26109 raft_consensus.cc:348] T 9c6408e9acee46adacd2bbd81669f27a P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } }
I20260807 11:09:32.431818 26109 raft_consensus.cc:374] T 9c6408e9acee46adacd2bbd81669f27a P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:32.432283 26109 raft_consensus.cc:729] T 9c6408e9acee46adacd2bbd81669f27a P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:32.433776 26109 consensus_queue.cc:260] T 9c6408e9acee46adacd2bbd81669f27a P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } }
I20260807 11:09:32.439517 26110 ts_tablet_manager.cc:1428] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.135s user 0.010s sys 0.000s
I20260807 11:09:32.440804 26110 tablet_bootstrap.cc:492] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:32.411931 26195 consensus_queue.cc:237] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:32.451105 26115 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:32.457240 26109 ts_tablet_manager.cc:1428] T 9c6408e9acee46adacd2bbd81669f27a P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.078s user 0.005s sys 0.005s
I20260807 11:09:32.460444 26108 tablet_bootstrap.cc:492] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:32.463841 26109 tablet_bootstrap.cc:492] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:32.516273 26208 consensus_queue.cc:1035] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:32.458403 26115 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } }
I20260807 11:09:32.572733 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:32.577188 26108 ts_tablet_manager.cc:1397] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.453s user 0.047s sys 0.097s
W20260807 11:09:32.585438 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 63867.3us
I20260807 11:09:32.589722 26115 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195)
I20260807 11:09:32.588346 26180 consensus_queue.cc:1035] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260807 11:09:32.608666 26033 raft_consensus.cc:3037] T 9c6408e9acee46adacd2bbd81669f27a P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:32.616346 26108 raft_consensus.cc:348] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:32.617429 25752 consensus_peers.cc:487] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 -> Peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Couldn't send request to peer b22d69db45194908ad3d345eba9badd9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:32.625633 26198 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:32.616912 26108 raft_consensus.cc:374] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:32.627928 26108 raft_consensus.cc:729] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:32.627512 26041 raft_consensus.cc:2379] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 1: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:09:32.628553 26108 consensus_queue.cc:260] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:32.633739 26280 consensus_queue.cc:1035] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260807 11:09:32.606743 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:32.658497 25981 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:09:32.675135 26108 ts_tablet_manager.cc:1428] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.097s user 0.005s sys 0.007s
I20260807 11:09:32.681807 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:32.682818 25808 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:32.679001 26171 consensus_queue.cc:1035] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260807 11:09:32.689291 26108 tablet_bootstrap.cc:492] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:32.695878 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:32.702687 26224 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:32.692555 26198 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:32.699807 26171 consensus_queue.cc:1035] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.031s
I20260807 11:09:32.763553 25660 catalog_manager.cc:5495] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 reported cstate change: term changed from 0 to 1, leader changed from <none> to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:32.760219 25926 raft_consensus.cc:2379] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 1: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:09:32.821269 26120 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:09:32.856082 26282 log.cc:924] Time spent T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.158s user 0.002s sys 0.000s
I20260807 11:09:32.870357 26110 tablet_bootstrap.cc:654] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:32.847913 26120 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:32.876865 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:32.879762 26120 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:32.881090 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:32.886173 26216 raft_consensus.cc:2726] T 9c6408e9acee46adacd2bbd81669f27a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260807 11:09:32.864399 26285 log.cc:924] Time spent T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.169s user 0.000s sys 0.001s
W20260807 11:09:32.883284 25750 leader_election.cc:341] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:32.894989 26224 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:32.903609 26280 raft_consensus.cc:482] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:32.904387 26280 raft_consensus.cc:504] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:32.921520 26280 leader_election.cc:288] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:32.923111 26108 tablet_bootstrap.cc:654] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:33.038444 26109 tablet_bootstrap.cc:700] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9: Time spent opening tablet: real 0.114s user 0.002s sys 0.000s
I20260807 11:09:33.040325 26109 tablet_bootstrap.cc:654] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:33.057520 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:09:33.063606 25869 leader_election.cc:341] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:33.090688 25808 raft_consensus.cc:2454] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:09:33.093044 25981 leader_election.cc:302] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:33.094673 26110 tablet_bootstrap.cc:492] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:33.095610 26302 raft_consensus.cc:2781] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:33.096252 26302 raft_consensus.cc:686] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:09:33.097368 26302 consensus_queue.cc:237] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.102816 26120 consensus_queue.cc:1035] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:33.104110 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:33.104935 26039 raft_consensus.cc:2454] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 0.
I20260807 11:09:33.114763 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:33.127998 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:33.129837 26110 ts_tablet_manager.cc:1397] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.689s user 0.013s sys 0.017s
I20260807 11:09:33.130627 26108 tablet_bootstrap.cc:492] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:33.131235 26108 ts_tablet_manager.cc:1397] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.452s user 0.012s sys 0.052s
W20260807 11:09:33.133280 25750 consensus_peers.cc:487] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 -> Peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Couldn't send request to peer 4fd6381544574ddca559ce63b9243efc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:33.134994 26108 raft_consensus.cc:348] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.135952 26108 raft_consensus.cc:374] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:33.136310 26108 raft_consensus.cc:729] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:33.139030 26110 raft_consensus.cc:348] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.139829 26110 raft_consensus.cc:374] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:33.140107 26110 raft_consensus.cc:729] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:33.150274 25752 leader_election.cc:302] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:33.151391 26120 raft_consensus.cc:2781] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:33.151809 26120 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:33.152134 26120 raft_consensus.cc:3037] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:33.174269 25872 leader_election.cc:302] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:33.182904 26280 raft_consensus.cc:2726] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260807 11:09:33.185135 25872 leader_election.cc:341] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:33.185727 25872 leader_election.cc:302] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:33.186975 26280 raft_consensus.cc:2726] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:33.195798 26120 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.217931 26108 consensus_queue.cc:260] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.220252 26110 consensus_queue.cc:260] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.221644 26176 consensus_queue.cc:1035] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:33.223068 26110 ts_tablet_manager.cc:1428] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.093s user 0.029s sys 0.050s
I20260807 11:09:33.223598 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:33.224666 25932 raft_consensus.cc:3037] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:33.224814 26110 tablet_bootstrap.cc:492] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:33.230430 26108 ts_tablet_manager.cc:1428] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.099s user 0.001s sys 0.009s
I20260807 11:09:33.231887 26108 tablet_bootstrap.cc:492] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:33.235976 26109 tablet_bootstrap.cc:492] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:33.236712 26109 ts_tablet_manager.cc:1397] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.778s user 0.044s sys 0.018s
W20260807 11:09:33.243968 25723 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 11:09:33.218119 26120 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:33.240150 26109 raft_consensus.cc:348] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.245847 26109 raft_consensus.cc:374] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:33.246042 26109 raft_consensus.cc:729] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:33.247188 26109 consensus_queue.cc:260] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.252414 26109 ts_tablet_manager.cc:1428] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260807 11:09:33.253700 26109 tablet_bootstrap.cc:492] T e702bb04fc98457f9d8ee704fcaeafcf P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:33.259763 26108 tablet_bootstrap.cc:654] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:33.261432 26110 tablet_bootstrap.cc:654] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:33.262439 25932 raft_consensus.cc:2454] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:09:33.262583 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:33.263320 26042 raft_consensus.cc:3037] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:33.264521 26109 tablet_bootstrap.cc:654] T e702bb04fc98457f9d8ee704fcaeafcf P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:33.274304 25750 leader_election.cc:302] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:09:33.276885 26120 raft_consensus.cc:2781] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:33.277397 26120 raft_consensus.cc:686] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:09:33.278223 26120 consensus_queue.cc:237] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.279887 26042 raft_consensus.cc:2454] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:09:33.293547 25808 raft_consensus.cc:3037] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:33.351778 26110 tablet_bootstrap.cc:492] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:33.350925 25660 catalog_manager.cc:5495] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260807 11:09:33.352483 26110 ts_tablet_manager.cc:1397] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.128s user 0.028s sys 0.044s
I20260807 11:09:33.357949 26110 raft_consensus.cc:348] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.358994 26110 raft_consensus.cc:374] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:33.359402 26110 raft_consensus.cc:729] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:33.359848 26109 tablet_bootstrap.cc:492] T e702bb04fc98457f9d8ee704fcaeafcf P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:33.360459 26109 ts_tablet_manager.cc:1397] T e702bb04fc98457f9d8ee704fcaeafcf P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.107s user 0.008s sys 0.012s
I20260807 11:09:33.360357 26110 consensus_queue.cc:260] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.364060 26110 ts_tablet_manager.cc:1428] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.011s user 0.006s sys 0.002s
I20260807 11:09:33.365773 26110 tablet_bootstrap.cc:492] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:33.366351 26109 raft_consensus.cc:348] T e702bb04fc98457f9d8ee704fcaeafcf P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.367003 26109 raft_consensus.cc:374] T e702bb04fc98457f9d8ee704fcaeafcf P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:33.367238 26109 raft_consensus.cc:729] T e702bb04fc98457f9d8ee704fcaeafcf P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:33.368315 26109 consensus_queue.cc:260] T e702bb04fc98457f9d8ee704fcaeafcf P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.369115 26302 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:33.369628 26302 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.370554 26109 ts_tablet_manager.cc:1428] T e702bb04fc98457f9d8ee704fcaeafcf P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20260807 11:09:33.371678 26109 tablet_bootstrap.cc:492] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:33.371764 26302 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:33.373626 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:33.373847 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:33.376019 25981 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:09:33.377400 26302 raft_consensus.cc:2726] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:33.380925 26110 tablet_bootstrap.cc:654] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:33.382519 26108 tablet_bootstrap.cc:492] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:33.385703 26108 ts_tablet_manager.cc:1397] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.154s user 0.045s sys 0.028s
I20260807 11:09:33.388170 26108 raft_consensus.cc:348] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.388891 26108 raft_consensus.cc:374] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:33.389163 26108 raft_consensus.cc:729] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:33.389894 26108 consensus_queue.cc:260] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.393533 26108 ts_tablet_manager.cc:1428] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.008s user 0.001s sys 0.004s
I20260807 11:09:33.397287 26110 tablet_bootstrap.cc:492] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:33.397706 26110 ts_tablet_manager.cc:1397] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.033s user 0.010s sys 0.000s
I20260807 11:09:33.398743 26108 tablet_bootstrap.cc:492] T 99e5feeda1eb4affb2532bda1b0055cd P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:33.400452 26109 tablet_bootstrap.cc:654] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:33.400234 26110 raft_consensus.cc:348] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.400943 26110 raft_consensus.cc:374] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:33.401258 26110 raft_consensus.cc:729] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:33.401949 26110 consensus_queue.cc:260] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.404170 26110 ts_tablet_manager.cc:1428] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260807 11:09:33.405714 26108 tablet_bootstrap.cc:654] T 99e5feeda1eb4affb2532bda1b0055cd P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:33.415445 26110 tablet_bootstrap.cc:492] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:33.422024 26110 tablet_bootstrap.cc:654] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:33.430406 25654 catalog_manager.cc:5495] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 reported cstate change: term changed from 0 to 1, leader changed from <none> to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:33.435242 26108 tablet_bootstrap.cc:492] T 99e5feeda1eb4affb2532bda1b0055cd P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:33.435722 26109 tablet_bootstrap.cc:492] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:33.435855 26108 ts_tablet_manager.cc:1397] T 99e5feeda1eb4affb2532bda1b0055cd P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.020s
I20260807 11:09:33.436156 26109 ts_tablet_manager.cc:1397] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.065s user 0.015s sys 0.007s
I20260807 11:09:33.438692 26109 raft_consensus.cc:348] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.439390 26109 raft_consensus.cc:374] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:33.439708 26109 raft_consensus.cc:729] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:33.440485 26180 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:33.440357 26109 consensus_queue.cc:260] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.440996 26180 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.447669 26109 ts_tablet_manager.cc:1428] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260807 11:09:33.448741 26109 tablet_bootstrap.cc:492] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:33.449781 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:33.450533 26033 raft_consensus.cc:2454] T e702bb04fc98457f9d8ee704fcaeafcf P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
I20260807 11:09:33.451303 26180 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:33.452965 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:33.454105 26110 tablet_bootstrap.cc:492] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:33.454999 25872 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:33.457641 26110 ts_tablet_manager.cc:1397] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.043s user 0.015s sys 0.008s
I20260807 11:09:33.458093 26224 raft_consensus.cc:2781] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:33.458498 26224 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:33.458844 26224 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:33.460444 26110 raft_consensus.cc:348] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.461149 26110 raft_consensus.cc:374] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:33.461410 26110 raft_consensus.cc:729] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:33.462023 26110 consensus_queue.cc:260] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.453704 25808 raft_consensus.cc:2454] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
I20260807 11:09:33.464109 26110 ts_tablet_manager.cc:1428] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260807 11:09:33.466034 26109 tablet_bootstrap.cc:654] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:33.469410 26110 tablet_bootstrap.cc:492] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:33.474519 26110 tablet_bootstrap.cc:654] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:33.479249 26109 tablet_bootstrap.cc:492] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:33.479619 26109 ts_tablet_manager.cc:1397] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.004s
I20260807 11:09:33.481675 26109 raft_consensus.cc:348] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.482331 26109 raft_consensus.cc:374] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:33.482585 26109 raft_consensus.cc:729] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:33.482800 26110 tablet_bootstrap.cc:492] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:33.483505 26110 ts_tablet_manager.cc:1397] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20260807 11:09:33.483201 26109 consensus_queue.cc:260] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.486409 26110 raft_consensus.cc:348] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.487150 26110 raft_consensus.cc:374] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:33.487444 26110 raft_consensus.cc:729] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:33.488158 26110 consensus_queue.cc:260] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.439119 26108 raft_consensus.cc:348] T 99e5feeda1eb4affb2532bda1b0055cd P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.489540 26108 raft_consensus.cc:374] T 99e5feeda1eb4affb2532bda1b0055cd P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:33.489846 26108 raft_consensus.cc:729] T 99e5feeda1eb4affb2532bda1b0055cd P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:33.490621 26110 ts_tablet_manager.cc:1428] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260807 11:09:33.490517 26108 consensus_queue.cc:260] T 99e5feeda1eb4affb2532bda1b0055cd P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.492010 26110 tablet_bootstrap.cc:492] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:33.492669 26108 ts_tablet_manager.cc:1428] T 99e5feeda1eb4affb2532bda1b0055cd P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.056s user 0.000s sys 0.003s
I20260807 11:09:33.494203 26109 ts_tablet_manager.cc:1428] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.014s user 0.006s sys 0.002s
I20260807 11:09:33.495534 26320 raft_consensus.cc:482] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:33.496011 26320 raft_consensus.cc:504] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.498204 26320 leader_election.cc:288] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:33.498852 26110 tablet_bootstrap.cc:654] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:33.503011 26042 raft_consensus.cc:3037] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:33.505239 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:33.506019 26109 tablet_bootstrap.cc:492] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
W20260807 11:09:33.506753 25752 leader_election.cc:341] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:33.513226 26195 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:33.513687 26195 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.515074 26109 tablet_bootstrap.cc:654] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:33.515820 26195 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:33.518702 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:09:33.520080 25752 leader_election.cc:341] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:33.523317 26224 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.526106 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:33.526721 25808 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:33.534298 25808 raft_consensus.cc:2454] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:09:33.535408 25869 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:09:33.536319 26180 raft_consensus.cc:2781] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:33.536839 26180 raft_consensus.cc:686] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:09:33.537719 26180 consensus_queue.cc:237] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.548568 26195 raft_consensus.cc:482] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:33.549146 26195 raft_consensus.cc:504] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.552594 26195 leader_election.cc:288] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:33.554163 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:33.554641 26302 consensus_queue.cc:1035] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260807 11:09:33.555593 25752 leader_election.cc:341] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:33.558576 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:33.559202 26033 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:33.574654 26224 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:33.581982 26033 raft_consensus.cc:2454] T e702bb04fc98457f9d8ee704fcaeafcf P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:09:33.588600 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:33.589321 25925 raft_consensus.cc:2454] T 99e5feeda1eb4affb2532bda1b0055cd P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 0.
I20260807 11:09:33.591148 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:33.593124 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:33.598194 25750 leader_election.cc:302] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
W20260807 11:09:33.599359 25750 leader_election.cc:341] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:33.599514 26195 raft_consensus.cc:2781] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:33.599872 26195 raft_consensus.cc:482] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:33.599771 25750 leader_election.cc:302] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:33.600507 26320 raft_consensus.cc:2726] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:33.604928 26115 consensus_queue.cc:1035] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260807 11:09:33.600211 26195 raft_consensus.cc:3037] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:33.612223 26110 tablet_bootstrap.cc:492] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:33.612715 26110 ts_tablet_manager.cc:1397] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.121s user 0.026s sys 0.050s
W20260807 11:09:33.609345 25750 leader_election.cc:341] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:33.621174 25750 leader_election.cc:302] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:33.621930 26120 raft_consensus.cc:2726] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:33.635313 26195 raft_consensus.cc:504] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.637938 26108 tablet_bootstrap.cc:492] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:33.646411 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:33.647013 25920 raft_consensus.cc:3037] T 99e5feeda1eb4affb2532bda1b0055cd P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:33.647699 26195 leader_election.cc:288] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:33.648685 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:09:33.650056 25752 leader_election.cc:341] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:33.652071 26110 raft_consensus.cc:348] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.652735 26110 raft_consensus.cc:374] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:33.653007 26110 raft_consensus.cc:729] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:33.653667 26110 consensus_queue.cc:260] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.658614 25658 catalog_manager.cc:5495] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:33.659528 26109 tablet_bootstrap.cc:492] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:33.660141 26109 ts_tablet_manager.cc:1397] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.154s user 0.056s sys 0.052s
I20260807 11:09:33.665693 26110 ts_tablet_manager.cc:1428] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.053s user 0.003s sys 0.003s
I20260807 11:09:33.671231 26110 tablet_bootstrap.cc:492] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:33.677131 26108 tablet_bootstrap.cc:654] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:33.676833 25920 raft_consensus.cc:2454] T 99e5feeda1eb4affb2532bda1b0055cd P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:09:33.684587 25750 leader_election.cc:302] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:09:33.685716 26120 raft_consensus.cc:2781] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:33.686486 26120 raft_consensus.cc:686] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:09:33.687779 26109 raft_consensus.cc:348] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.688557 26109 raft_consensus.cc:374] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:33.687887 26120 consensus_queue.cc:237] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.688846 26109 raft_consensus.cc:729] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:33.689543 26109 consensus_queue.cc:260] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.691028 26297 raft_consensus.cc:482] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:33.691453 26297 raft_consensus.cc:504] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.693785 26297 leader_election.cc:288] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:33.698660 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:33.699342 25808 raft_consensus.cc:2454] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 0.
I20260807 11:09:33.699927 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:33.700591 25981 leader_election.cc:302] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:33.701383 26297 raft_consensus.cc:2781] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260807 11:09:33.701556 25981 leader_election.cc:341] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:33.701730 26297 raft_consensus.cc:482] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:33.702060 26297 raft_consensus.cc:3037] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:33.698628 26110 tablet_bootstrap.cc:654] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:33.710039 26109 ts_tablet_manager.cc:1428] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.050s user 0.005s sys 0.001s
I20260807 11:09:33.711298 26109 tablet_bootstrap.cc:492] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:33.714929 26115 raft_consensus.cc:482] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:33.714810 26297 raft_consensus.cc:504] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.715345 26115 raft_consensus.cc:504] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.717120 26115 leader_election.cc:288] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:33.718284 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:33.718880 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:09:33.719756 25981 leader_election.cc:341] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
W20260807 11:09:33.720626 25981 leader_election.cc:341] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:33.721165 25981 leader_election.cc:302] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:09:33.722281 26115 raft_consensus.cc:2726] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:33.731155 26195 consensus_queue.cc:1035] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:33.739059 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:33.739794 25808 raft_consensus.cc:3037] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:33.741283 26297 leader_election.cc:288] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:33.742318 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:09:33.743639 25981 leader_election.cc:341] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:33.745689 25660 catalog_manager.cc:5495] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 reported cstate change: term changed from 0 to 1, leader changed from <none> to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:33.766606 26108 tablet_bootstrap.cc:492] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:33.767172 26108 ts_tablet_manager.cc:1397] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.129s user 0.018s sys 0.010s
I20260807 11:09:33.767738 26109 tablet_bootstrap.cc:654] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:33.772324 25808 raft_consensus.cc:2454] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:09:33.774930 26110 tablet_bootstrap.cc:492] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:33.775913 26110 ts_tablet_manager.cc:1397] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.105s user 0.020s sys 0.007s
I20260807 11:09:33.777311 25981 leader_election.cc:302] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:33.778559 26297 raft_consensus.cc:2781] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:33.779032 26176 consensus_queue.cc:1035] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:33.780025 26297 raft_consensus.cc:686] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:09:33.781229 26297 consensus_queue.cc:237] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.787356 26108 raft_consensus.cc:348] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.788230 26108 raft_consensus.cc:374] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:33.788616 26108 raft_consensus.cc:729] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:33.783864 26110 raft_consensus.cc:348] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.789467 26110 raft_consensus.cc:374] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:33.789785 26110 raft_consensus.cc:729] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:33.789565 26108 consensus_queue.cc:260] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.790437 26110 consensus_queue.cc:260] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.795454 26108 ts_tablet_manager.cc:1428] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.028s user 0.005s sys 0.001s
I20260807 11:09:33.799120 26109 tablet_bootstrap.cc:492] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:33.799602 26109 ts_tablet_manager.cc:1397] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.089s user 0.017s sys 0.009s
I20260807 11:09:33.801434 26176 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:33.801903 26176 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.804836 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:33.805310 26110 ts_tablet_manager.cc:1428] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.029s user 0.004s sys 0.003s
I20260807 11:09:33.806366 26110 tablet_bootstrap.cc:492] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
W20260807 11:09:33.806496 25750 leader_election.cc:341] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:33.807590 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:33.809334 26108 tablet_bootstrap.cc:492] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
W20260807 11:09:33.809276 25752 leader_election.cc:341] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:33.809883 25752 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:33.810858 26320 raft_consensus.cc:2726] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:33.804857 26176 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:33.812834 26109 raft_consensus.cc:348] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.813514 26109 raft_consensus.cc:374] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:33.814141 26109 raft_consensus.cc:729] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:33.814787 26109 consensus_queue.cc:260] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.816751 26109 ts_tablet_manager.cc:1428] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260807 11:09:33.817718 26109 tablet_bootstrap.cc:492] T f35a17e1683e41f4b146284f5cf09c58 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:33.822551 26108 tablet_bootstrap.cc:654] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:33.823120 26109 tablet_bootstrap.cc:654] T f35a17e1683e41f4b146284f5cf09c58 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:33.827215 26115 raft_consensus.cc:482] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:33.827739 26115 raft_consensus.cc:504] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.831086 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:09:33.832963 25981 leader_election.cc:341] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:33.836911 26115 leader_election.cc:288] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:33.834759 25660 catalog_manager.cc:5495] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260807 11:09:33.838625 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:33.839545 26176 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:33.839783 25919 raft_consensus.cc:2454] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 0.
I20260807 11:09:33.840034 26176 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.841284 25981 leader_election.cc:302] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:09:33.842185 26176 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:33.842500 26115 raft_consensus.cc:2781] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:33.842876 26115 raft_consensus.cc:482] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:33.843250 26115 raft_consensus.cc:3037] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:33.843874 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:09:33.845789 25752 leader_election.cc:341] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:33.846626 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:09:33.848474 25750 leader_election.cc:341] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:33.848918 25750 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:33.850109 26176 raft_consensus.cc:2726] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:33.842871 26110 tablet_bootstrap.cc:654] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:33.851703 26109 tablet_bootstrap.cc:492] T f35a17e1683e41f4b146284f5cf09c58 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:33.852252 26109 ts_tablet_manager.cc:1397] T f35a17e1683e41f4b146284f5cf09c58 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.035s user 0.011s sys 0.011s
I20260807 11:09:33.854732 26109 raft_consensus.cc:348] T f35a17e1683e41f4b146284f5cf09c58 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.855566 26109 raft_consensus.cc:374] T f35a17e1683e41f4b146284f5cf09c58 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:33.855809 26109 raft_consensus.cc:729] T f35a17e1683e41f4b146284f5cf09c58 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:33.856377 26109 consensus_queue.cc:260] T f35a17e1683e41f4b146284f5cf09c58 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.859299 26109 ts_tablet_manager.cc:1428] T f35a17e1683e41f4b146284f5cf09c58 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260807 11:09:33.861258 26109 tablet_bootstrap.cc:492] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:33.863585 26115 raft_consensus.cc:504] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.867909 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:33.866353 26115 leader_election.cc:288] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:33.868309 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:33.868952 25919 raft_consensus.cc:3037] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:33.870878 26109 tablet_bootstrap.cc:654] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:33.871402 26108 tablet_bootstrap.cc:492] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:33.871853 26108 ts_tablet_manager.cc:1397] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.073s user 0.002s sys 0.015s
W20260807 11:09:33.873139 25981 leader_election.cc:341] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:33.879559 26108 raft_consensus.cc:348] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.880085 26108 raft_consensus.cc:374] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:33.880308 26108 raft_consensus.cc:729] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:33.880914 26108 consensus_queue.cc:260] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.882826 26108 ts_tablet_manager.cc:1428] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20260807 11:09:33.886309 25919 raft_consensus.cc:2454] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:09:33.887837 25981 leader_election.cc:302] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:09:33.888430 26108 tablet_bootstrap.cc:492] T 986fe968b7be4eab9c480ec85c0384b0 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:33.892241 26180 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:33.892669 26180 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.894654 26180 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:33.896240 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:09:33.897907 25869 leader_election.cc:341] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:33.898662 26108 tablet_bootstrap.cc:654] T 986fe968b7be4eab9c480ec85c0384b0 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:33.899135 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:33.899832 26042 raft_consensus.cc:2454] T f35a17e1683e41f4b146284f5cf09c58 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
I20260807 11:09:33.901118 25872 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:09:33.902321 26180 raft_consensus.cc:2781] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:33.902714 26180 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:33.902768 26115 raft_consensus.cc:2781] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:33.903158 26180 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:33.904160 26115 raft_consensus.cc:686] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:09:33.904342 26109 tablet_bootstrap.cc:492] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:33.911285 26297 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:33.912357 26109 ts_tablet_manager.cc:1397] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.052s user 0.008s sys 0.004s
I20260807 11:09:33.912334 26297 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.914439 26297 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:33.914980 26109 raft_consensus.cc:348] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.915660 26109 raft_consensus.cc:374] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:33.915912 26109 raft_consensus.cc:729] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:33.916803 26110 tablet_bootstrap.cc:492] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:33.916502 26109 consensus_queue.cc:260] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.917277 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:33.917958 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:33.919158 26109 ts_tablet_manager.cc:1428] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260807 11:09:33.920250 25981 leader_election.cc:341] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:33.917263 26110 ts_tablet_manager.cc:1397] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.111s user 0.015s sys 0.014s
W20260807 11:09:33.921084 25981 leader_election.cc:341] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:33.921633 25981 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:09:33.923202 26297 raft_consensus.cc:2726] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:33.924371 26110 raft_consensus.cc:348] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.925140 26110 raft_consensus.cc:374] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:33.925423 26110 raft_consensus.cc:729] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:33.924546 26115 consensus_queue.cc:237] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.918903 26180 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.920439 26109 tablet_bootstrap.cc:492] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:33.929271 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
W20260807 11:09:33.930866 25869 leader_election.cc:341] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:33.932829 26180 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:33.933115 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:33.933823 26042 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:33.933972 26110 consensus_queue.cc:260] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.937592 26110 ts_tablet_manager.cc:1428] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260807 11:09:33.948498 26109 tablet_bootstrap.cc:654] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:33.949043 26042 raft_consensus.cc:2454] T f35a17e1683e41f4b146284f5cf09c58 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:09:33.953791 25872 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:09:33.952121 26108 tablet_bootstrap.cc:492] T 986fe968b7be4eab9c480ec85c0384b0 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:33.955766 26108 ts_tablet_manager.cc:1397] T 986fe968b7be4eab9c480ec85c0384b0 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.068s user 0.039s sys 0.010s
I20260807 11:09:33.956894 26280 raft_consensus.cc:2781] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:33.957394 26280 raft_consensus.cc:686] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:09:33.958276 26108 raft_consensus.cc:348] T 986fe968b7be4eab9c480ec85c0384b0 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.958967 26108 raft_consensus.cc:374] T 986fe968b7be4eab9c480ec85c0384b0 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:33.959209 26108 raft_consensus.cc:729] T 986fe968b7be4eab9c480ec85c0384b0 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:33.959786 26108 consensus_queue.cc:260] T 986fe968b7be4eab9c480ec85c0384b0 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.961797 26108 ts_tablet_manager.cc:1428] T 986fe968b7be4eab9c480ec85c0384b0 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260807 11:09:33.962644 26108 tablet_bootstrap.cc:492] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:33.958154 26280 consensus_queue.cc:237] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.963423 26110 tablet_bootstrap.cc:492] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:33.968582 26109 tablet_bootstrap.cc:492] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:33.969080 26109 ts_tablet_manager.cc:1397] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.049s user 0.007s sys 0.008s
I20260807 11:09:33.969587 25660 catalog_manager.cc:5495] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260807 11:09:33.972668 26108 tablet_bootstrap.cc:654] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:33.972859 26110 tablet_bootstrap.cc:654] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:33.978428 26109 raft_consensus.cc:348] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.979175 26109 raft_consensus.cc:374] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:33.979460 26109 raft_consensus.cc:729] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:33.980165 26109 consensus_queue.cc:260] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:33.982749 26109 ts_tablet_manager.cc:1428] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260807 11:09:33.984200 26109 tablet_bootstrap.cc:492] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:33.990487 26109 tablet_bootstrap.cc:654] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:33.996397 25658 catalog_manager.cc:5495] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:34.006044 26297 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:34.007335 26297 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.010546 26176 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:34.011377 26176 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.014549 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:09:34.016834 25981 leader_election.cc:341] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:34.020638 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:34.026172 26180 raft_consensus.cc:482] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:34.026325 26176 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:34.026805 26180 raft_consensus.cc:504] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:34.028112 25981 leader_election.cc:341] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:34.028927 25981 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:09:34.031364 26115 raft_consensus.cc:2726] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:34.032261 26297 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:34.033704 26180 leader_election.cc:288] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:34.041249 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:09:34.043815 25869 leader_election.cc:341] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:34.035514 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:34.045486 26042 raft_consensus.cc:2454] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 0.
I20260807 11:09:34.045310 26280 consensus_queue.cc:1035] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:34.035764 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:34.046862 26039 raft_consensus.cc:2454] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
I20260807 11:09:34.048722 25752 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:34.050118 26320 raft_consensus.cc:2781] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:34.050487 26320 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:34.050938 26320 raft_consensus.cc:3037] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:34.052270 25872 leader_election.cc:302] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:09:34.036696 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:09:34.057129 25750 leader_election.cc:341] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:34.057879 26297 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:34.058633 26297 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.061165 26280 raft_consensus.cc:2781] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:34.062310 26280 raft_consensus.cc:482] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:34.063324 26280 raft_consensus.cc:3037] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:34.094755 26180 consensus_queue.cc:1035] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:34.121193 25919 raft_consensus.cc:3037] T 986fe968b7be4eab9c480ec85c0384b0 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:34.154983 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:09:34.158418 25981 leader_election.cc:341] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
W20260807 11:09:34.180600 25752 consensus_peers.cc:487] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 -> Peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Couldn't send request to peer b22d69db45194908ad3d345eba9badd9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:34.192590 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:09:34.195369 25981 leader_election.cc:341] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:34.196177 25981 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:09:34.197896 26115 raft_consensus.cc:2726] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:34.220376 26297 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:34.229652 26176 consensus_queue.cc:1035] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:34.245322 26110 tablet_bootstrap.cc:492] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:34.245911 26108 tablet_bootstrap.cc:492] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:34.245991 26110 ts_tablet_manager.cc:1397] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.283s user 0.078s sys 0.098s
I20260807 11:09:34.246515 26108 ts_tablet_manager.cc:1397] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.284s user 0.033s sys 0.078s
I20260807 11:09:34.266460 26109 tablet_bootstrap.cc:492] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:34.267104 26109 ts_tablet_manager.cc:1397] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.283s user 0.045s sys 0.054s
I20260807 11:09:34.269564 26280 raft_consensus.cc:504] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.272405 26110 raft_consensus.cc:348] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.274065 26110 raft_consensus.cc:374] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:34.274516 26110 raft_consensus.cc:729] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:34.275434 26110 consensus_queue.cc:260] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.275316 26378 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:09:34.280280 26378 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.285580 26109 raft_consensus.cc:348] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.286264 26109 raft_consensus.cc:374] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:34.286509 26109 raft_consensus.cc:729] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:34.287115 26109 consensus_queue.cc:260] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.273766 26320 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.294462 26108 raft_consensus.cc:348] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.295305 26108 raft_consensus.cc:374] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:34.295603 26108 raft_consensus.cc:729] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:34.300717 26115 consensus_queue.cc:1035] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:34.308300 26280 leader_election.cc:288] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:34.309908 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:34.310402 26108 consensus_queue.cc:260] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.311097 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:34.318178 26320 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:34.318488 26042 raft_consensus.cc:3037] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:34.319707 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:09:34.315649 25869 leader_election.cc:341] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:34.317040 26110 ts_tablet_manager.cc:1428] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.071s user 0.009s sys 0.000s
I20260807 11:09:34.322017 26110 tablet_bootstrap.cc:492] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:34.322651 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:34.326328 26039 raft_consensus.cc:3037] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
W20260807 11:09:34.329144 25981 consensus_peers.cc:487] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 -> Peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Couldn't send request to peer 4fd6381544574ddca559ce63b9243efc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:34.330963 26378 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:34.336539 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:34.340281 26110 tablet_bootstrap.cc:654] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:34.340533 26109 ts_tablet_manager.cc:1428] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.073s user 0.009s sys 0.000s
I20260807 11:09:34.341672 26109 tablet_bootstrap.cc:492] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:34.343528 25921 raft_consensus.cc:3037] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:34.350536 26108 ts_tablet_manager.cc:1428] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.104s user 0.002s sys 0.007s
I20260807 11:09:34.349879 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:34.351855 26108 tablet_bootstrap.cc:492] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:34.353189 25869 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:34.355786 26378 raft_consensus.cc:2726] T 986fe968b7be4eab9c480ec85c0384b0 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 11:09:34.375069 25869 consensus_peers.cc:487] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc -> Peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Couldn't send request to peer 606d49e053a346e486f48cb31832b815. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260807 11:09:34.361716 25723 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 11:09:34.430034 26042 raft_consensus.cc:2454] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:09:34.438936 26039 raft_consensus.cc:2454] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:09:34.441787 25872 leader_election.cc:302] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:09:34.443434 26110 tablet_bootstrap.cc:492] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:34.443981 26110 ts_tablet_manager.cc:1397] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.122s user 0.005s sys 0.014s
I20260807 11:09:34.443800 26224 raft_consensus.cc:2781] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:34.444720 26224 raft_consensus.cc:686] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:09:34.446283 26224 consensus_queue.cc:237] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.448045 26110 raft_consensus.cc:348] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.448559 26110 raft_consensus.cc:374] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:34.448817 26110 raft_consensus.cc:729] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:34.449451 26110 consensus_queue.cc:260] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.454736 26110 ts_tablet_manager.cc:1428] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260807 11:09:34.459761 25752 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:34.459647 26302 consensus_queue.cc:1035] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260807 11:09:34.469444 25921 raft_consensus.cc:2454] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:09:34.470088 26176 raft_consensus.cc:2781] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:34.471823 26108 tablet_bootstrap.cc:654] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:34.457332 26110 tablet_bootstrap.cc:492] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:34.478703 26176 raft_consensus.cc:686] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:09:34.484141 26176 consensus_queue.cc:237] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.505728 26109 tablet_bootstrap.cc:654] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:34.508846 26110 tablet_bootstrap.cc:654] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:34.517088 25806 raft_consensus.cc:3037] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:34.530633 25654 catalog_manager.cc:5495] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:34.543414 26108 tablet_bootstrap.cc:492] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:34.543991 26108 ts_tablet_manager.cc:1397] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.192s user 0.011s sys 0.015s
I20260807 11:09:34.545037 26297 consensus_queue.cc:1035] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:34.552358 26176 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:34.553231 26176 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.557938 26108 raft_consensus.cc:348] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.558794 26108 raft_consensus.cc:374] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:34.558570 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:34.559084 26108 raft_consensus.cc:729] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
W20260807 11:09:34.560349 25750 leader_election.cc:341] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:34.561285 26108 consensus_queue.cc:260] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.562126 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:34.562855 26042 raft_consensus.cc:2454] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 0.
I20260807 11:09:34.564502 25752 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:34.566830 26389 raft_consensus.cc:2781] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:34.567328 26389 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:34.566584 25660 catalog_manager.cc:5495] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 reported cstate change: term changed from 0 to 1, leader changed from <none> to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:34.567804 26389 raft_consensus.cc:3037] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:34.569762 26176 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:34.588538 26108 ts_tablet_manager.cc:1428] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.044s user 0.009s sys 0.001s
I20260807 11:09:34.589107 26110 tablet_bootstrap.cc:492] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:34.589669 26110 ts_tablet_manager.cc:1397] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.133s user 0.019s sys 0.001s
I20260807 11:09:34.589794 26108 tablet_bootstrap.cc:492] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:34.601442 26110 raft_consensus.cc:348] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.602293 26110 raft_consensus.cc:374] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:34.602859 26110 raft_consensus.cc:729] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:34.603647 26110 consensus_queue.cc:260] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.603964 26180 consensus_queue.cc:1035] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:34.606467 26110 ts_tablet_manager.cc:1428] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.016s user 0.007s sys 0.001s
I20260807 11:09:34.607034 26108 tablet_bootstrap.cc:654] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:34.607477 26110 tablet_bootstrap.cc:492] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:34.613615 26297 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:34.614358 26297 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.615293 26109 tablet_bootstrap.cc:492] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:34.615937 26109 ts_tablet_manager.cc:1397] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.274s user 0.041s sys 0.050s
I20260807 11:09:34.619037 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:34.620095 26109 raft_consensus.cc:348] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.620920 26109 raft_consensus.cc:374] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:34.621238 26109 raft_consensus.cc:729] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
W20260807 11:09:34.621534 25981 leader_election.cc:341] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:34.622123 26109 consensus_queue.cc:260] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.627957 26109 ts_tablet_manager.cc:1428] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.012s user 0.009s sys 0.000s
I20260807 11:09:34.629292 26109 tablet_bootstrap.cc:492] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:34.623958 26389 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.632936 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:34.641196 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:09:34.644258 25981 leader_election.cc:341] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:34.647712 25981 leader_election.cc:302] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
W20260807 11:09:34.649857 25750 leader_election.cc:341] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:34.649732 26297 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:34.649269 26302 raft_consensus.cc:2726] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:34.648825 26110 tablet_bootstrap.cc:654] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:34.666534 26389 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:34.666901 26109 tablet_bootstrap.cc:654] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:34.680378 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:34.685091 26042 raft_consensus.cc:3037] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:34.753283 26108 tablet_bootstrap.cc:492] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:34.753963 26108 ts_tablet_manager.cc:1397] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.164s user 0.015s sys 0.033s
I20260807 11:09:34.759322 26108 raft_consensus.cc:348] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.760387 26108 raft_consensus.cc:374] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:34.760743 26108 raft_consensus.cc:729] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:34.765590 26110 tablet_bootstrap.cc:492] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:34.766338 26110 ts_tablet_manager.cc:1397] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.159s user 0.008s sys 0.011s
I20260807 11:09:34.761782 26108 consensus_queue.cc:260] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.772116 26109 tablet_bootstrap.cc:492] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:34.772702 26109 ts_tablet_manager.cc:1397] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.144s user 0.011s sys 0.016s
I20260807 11:09:34.774781 26110 raft_consensus.cc:348] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.775665 26110 raft_consensus.cc:374] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:34.775982 26110 raft_consensus.cc:729] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:34.777225 26110 consensus_queue.cc:260] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.780061 26042 raft_consensus.cc:2454] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:09:34.780656 26109 raft_consensus.cc:348] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.781280 26109 raft_consensus.cc:374] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:34.781459 26109 raft_consensus.cc:729] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:34.781531 25752 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:34.782246 26109 consensus_queue.cc:260] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.783723 26109 ts_tablet_manager.cc:1428] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20260807 11:09:34.784503 26109 tablet_bootstrap.cc:492] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:34.786780 26110 ts_tablet_manager.cc:1428] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.020s user 0.003s sys 0.005s
I20260807 11:09:34.789161 26110 tablet_bootstrap.cc:492] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:34.795336 26110 tablet_bootstrap.cc:654] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:34.795405 26108 ts_tablet_manager.cc:1428] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.041s user 0.009s sys 0.000s
I20260807 11:09:34.797012 26109 tablet_bootstrap.cc:654] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:34.798417 26389 raft_consensus.cc:2781] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:34.798995 26389 raft_consensus.cc:686] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:09:34.799850 26389 consensus_queue.cc:237] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.805945 26108 tablet_bootstrap.cc:492] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:34.821879 25654 catalog_manager.cc:5495] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 reported cstate change: term changed from 0 to 1, leader changed from <none> to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:34.834190 26110 tablet_bootstrap.cc:492] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:34.834728 26110 ts_tablet_manager.cc:1397] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.046s user 0.009s sys 0.004s
I20260807 11:09:34.835721 26109 tablet_bootstrap.cc:492] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:34.836498 26109 ts_tablet_manager.cc:1397] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.052s user 0.020s sys 0.006s
I20260807 11:09:34.838996 26110 raft_consensus.cc:348] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.839825 26110 raft_consensus.cc:374] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:34.840199 26110 raft_consensus.cc:729] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:34.840435 26108 tablet_bootstrap.cc:654] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:34.840998 26110 consensus_queue.cc:260] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.847694 26109 raft_consensus.cc:348] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.848608 26109 raft_consensus.cc:374] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:34.849097 26109 raft_consensus.cc:729] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:34.850349 26110 ts_tablet_manager.cc:1428] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.015s user 0.007s sys 0.002s
I20260807 11:09:34.850179 26109 consensus_queue.cc:260] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.851672 26110 tablet_bootstrap.cc:492] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:34.853588 26109 ts_tablet_manager.cc:1428] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260807 11:09:34.855970 26109 tablet_bootstrap.cc:492] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:34.875007 26109 tablet_bootstrap.cc:654] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:34.880112 26108 tablet_bootstrap.cc:492] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:34.880676 26108 ts_tablet_manager.cc:1397] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.075s user 0.018s sys 0.015s
I20260807 11:09:34.882510 26110 tablet_bootstrap.cc:654] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:34.884264 26389 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:34.885066 26389 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.885480 26108 raft_consensus.cc:348] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.886243 26108 raft_consensus.cc:374] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:34.886510 26108 raft_consensus.cc:729] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:34.887225 26108 consensus_queue.cc:260] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.888679 26180 consensus_queue.cc:1035] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:34.891561 26390 consensus_queue.cc:1035] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:34.893491 26108 ts_tablet_manager.cc:1428] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.012s user 0.007s sys 0.001s
I20260807 11:09:34.896338 26302 raft_consensus.cc:482] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:34.896229 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:34.897220 26302 raft_consensus.cc:504] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.898995 26390 raft_consensus.cc:482] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:34.899693 26390 raft_consensus.cc:504] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.888319 26389 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:34.903671 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:34.905004 26108 tablet_bootstrap.cc:492] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:34.905189 26390 leader_election.cc:288] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:34.906648 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:34.905970 26390 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:34.909233 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:34.909173 26390 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:34.909860 25981 leader_election.cc:341] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:34.915208 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:34.922151 25752 leader_election.cc:302] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:34.916749 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:34.933079 26180 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:34.939368 26390 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:34.917415 25750 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:34.940559 26389 raft_consensus.cc:2726] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:34.943086 26280 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:34.943841 26280 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:34.944800 25750 leader_election.cc:341] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:34.943871 26389 raft_consensus.cc:2726] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:34.947145 26280 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:34.947753 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:34.948220 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:34.948650 26042 raft_consensus.cc:2454] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 0.
I20260807 11:09:34.939836 26180 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:34.951493 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:34.949239 25806 raft_consensus.cc:2454] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
I20260807 11:09:34.953890 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:34.954649 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:34.952430 26042 raft_consensus.cc:2454] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
I20260807 11:09:34.954720 25808 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
I20260807 11:09:34.918987 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:34.919175 26302 leader_election.cc:288] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:34.957204 25869 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:09:34.953300 26180 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:09:34.958810 25872 leader_election.cc:341] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:34.959334 25869 leader_election.cc:302] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:09:34.958801 26280 raft_consensus.cc:2781] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:34.960664 26280 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:34.961301 26280 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:34.960637 26180 raft_consensus.cc:2781] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:34.966951 26180 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:34.967478 26180 raft_consensus.cc:3037] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:34.977381 25752 leader_election.cc:302] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:34.975358 25806 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:34.983299 26176 raft_consensus.cc:2781] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:34.983858 26176 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:34.984310 26176 raft_consensus.cc:3037] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
W20260807 11:09:34.987628 25981 leader_election.cc:341] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:34.987177 25808 raft_consensus.cc:3037] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:34.988464 25981 leader_election.cc:302] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:09:35.007411 26302 raft_consensus.cc:2726] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:35.024541 26378 consensus_queue.cc:1035] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:35.025710 26280 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.030499 26109 tablet_bootstrap.cc:492] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:35.031028 26109 ts_tablet_manager.cc:1397] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.175s user 0.027s sys 0.012s
I20260807 11:09:35.032234 26110 tablet_bootstrap.cc:492] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:35.032757 26110 ts_tablet_manager.cc:1397] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.181s user 0.015s sys 0.013s
I20260807 11:09:35.044608 26180 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.045419 26110 raft_consensus.cc:348] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.046283 26110 raft_consensus.cc:374] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:35.049242 26110 raft_consensus.cc:729] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:35.049665 26110 consensus_queue.cc:260] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.050993 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:35.052018 25803 raft_consensus.cc:3037] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:35.053464 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:35.054466 26033 raft_consensus.cc:3037] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:35.041967 26176 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.057549 26302 raft_consensus.cc:482] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:35.062075 26302 raft_consensus.cc:504] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.064782 26176 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:35.067303 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:35.073951 26280 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:35.076326 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:35.074904 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:35.056376 26180 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:09:35.080220 25750 leader_election.cc:341] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:35.139338 26108 tablet_bootstrap.cc:654] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:35.140946 25801 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
W20260807 11:09:35.152086 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 92327.4us
I20260807 11:09:35.154704 26390 consensus_queue.cc:1035] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260807 11:09:35.141848 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 61707.5us
I20260807 11:09:35.163863 26110 ts_tablet_manager.cc:1428] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.131s user 0.007s sys 0.000s
W20260807 11:09:35.166546 25872 leader_election.cc:341] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:35.167078 25803 raft_consensus.cc:2454] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:09:35.166447 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:35.162653 26302 leader_election.cc:288] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:35.169220 25869 leader_election.cc:302] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:09:35.169569 26042 raft_consensus.cc:3037] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:35.170194 26422 raft_consensus.cc:2781] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:35.170687 26422 raft_consensus.cc:686] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:09:35.174463 26110 tablet_bootstrap.cc:492] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:35.184993 25801 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:09:35.185534 26422 consensus_queue.cc:237] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.187085 25869 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:09:35.190256 26378 raft_consensus.cc:482] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:35.196276 26378 raft_consensus.cc:504] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.200399 26280 raft_consensus.cc:2781] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:35.204780 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:35.205881 25797 raft_consensus.cc:2454] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
I20260807 11:09:35.201687 26280 raft_consensus.cc:686] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:09:35.208372 25869 leader_election.cc:302] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:09:35.210137 26180 raft_consensus.cc:2781] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:35.210748 26180 raft_consensus.cc:482] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:35.211261 26180 raft_consensus.cc:3037] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:35.212297 26109 raft_consensus.cc:348] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.212906 26109 raft_consensus.cc:374] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:35.213198 26109 raft_consensus.cc:729] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:35.213902 26109 consensus_queue.cc:260] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.217729 26280 consensus_queue.cc:237] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.220530 26109 ts_tablet_manager.cc:1428] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.189s user 0.007s sys 0.002s
I20260807 11:09:35.221879 26109 tablet_bootstrap.cc:492] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:35.221483 26378 leader_election.cc:288] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:35.228546 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:09:35.234211 25981 leader_election.cc:341] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:35.234571 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:35.239013 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:35.242561 26033 raft_consensus.cc:2454] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
W20260807 11:09:35.244105 25872 leader_election.cc:341] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:35.258453 26110 tablet_bootstrap.cc:654] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:35.251518 26042 raft_consensus.cc:2454] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
W20260807 11:09:35.258984 25981 leader_election.cc:341] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:35.271883 25752 leader_election.cc:302] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:35.280887 26180 raft_consensus.cc:504] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.284488 26390 raft_consensus.cc:2781] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:35.286770 26109 tablet_bootstrap.cc:654] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:35.289862 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:35.286715 26390 raft_consensus.cc:686] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:09:35.289063 26108 tablet_bootstrap.cc:492] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:35.291119 26108 ts_tablet_manager.cc:1397] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.386s user 0.009s sys 0.026s
W20260807 11:09:35.291742 25872 leader_election.cc:341] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:35.295502 26108 raft_consensus.cc:348] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.296361 26108 raft_consensus.cc:374] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:35.297540 26108 raft_consensus.cc:729] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:35.298430 26108 consensus_queue.cc:260] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.297401 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:35.296957 25654 catalog_manager.cc:5495] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:35.303201 26108 ts_tablet_manager.cc:1428] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.012s user 0.007s sys 0.001s
I20260807 11:09:35.300890 25808 raft_consensus.cc:3037] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:35.280049 25981 leader_election.cc:302] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:09:35.305665 26426 raft_consensus.cc:2726] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:35.299914 26180 leader_election.cc:288] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:35.307420 26390 consensus_queue.cc:237] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:35.324801 25750 consensus_peers.cc:487] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 -> Peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Couldn't send request to peer 4fd6381544574ddca559ce63b9243efc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:35.327061 26108 tablet_bootstrap.cc:492] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:35.329463 26110 tablet_bootstrap.cc:492] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:35.329890 26110 ts_tablet_manager.cc:1397] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.156s user 0.018s sys 0.011s
I20260807 11:09:35.320466 25808 raft_consensus.cc:2454] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
W20260807 11:09:35.327245 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 68872.8us
I20260807 11:09:35.360446 25869 leader_election.cc:302] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:09:35.362525 26180 raft_consensus.cc:2781] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:35.363278 26180 raft_consensus.cc:686] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:09:35.365259 26180 consensus_queue.cc:237] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.366369 25654 catalog_manager.cc:5495] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:35.420324 26110 raft_consensus.cc:348] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.421334 26110 raft_consensus.cc:374] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:35.421746 26110 raft_consensus.cc:729] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:35.422505 26110 consensus_queue.cc:260] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.424753 26110 ts_tablet_manager.cc:1428] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.095s user 0.027s sys 0.053s
I20260807 11:09:35.426106 26110 tablet_bootstrap.cc:492] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:35.433188 26110 tablet_bootstrap.cc:654] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:35.444016 25658 catalog_manager.cc:5495] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 reported cstate change: term changed from 0 to 1, leader changed from <none> to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:35.460275 26109 tablet_bootstrap.cc:492] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:35.460877 26109 ts_tablet_manager.cc:1397] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.239s user 0.015s sys 0.008s
I20260807 11:09:35.460942 26390 consensus_queue.cc:1035] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:09:35.463431 26110 tablet_bootstrap.cc:492] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:35.463903 26110 ts_tablet_manager.cc:1397] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.038s user 0.011s sys 0.000s
I20260807 11:09:35.466312 26110 raft_consensus.cc:348] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.467000 26110 raft_consensus.cc:374] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:35.467259 26110 raft_consensus.cc:729] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:35.467934 26110 consensus_queue.cc:260] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.470327 26109 raft_consensus.cc:348] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.470701 26110 ts_tablet_manager.cc:1428] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260807 11:09:35.471408 26109 raft_consensus.cc:374] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:35.472193 26109 raft_consensus.cc:729] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:35.472293 26110 tablet_bootstrap.cc:492] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:35.479754 26109 consensus_queue.cc:260] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.481894 26110 tablet_bootstrap.cc:654] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:35.482030 26109 ts_tablet_manager.cc:1428] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260807 11:09:35.476585 25654 catalog_manager.cc:5495] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:35.483429 26109 tablet_bootstrap.cc:492] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:35.440606 26108 tablet_bootstrap.cc:654] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:35.497380 26110 tablet_bootstrap.cc:492] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:35.497848 26110 ts_tablet_manager.cc:1397] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.008s
I20260807 11:09:35.501701 26110 raft_consensus.cc:348] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.502444 26110 raft_consensus.cc:374] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:35.502724 26110 raft_consensus.cc:729] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:35.503350 26110 consensus_queue.cc:260] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.507149 26110 ts_tablet_manager.cc:1428] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260807 11:09:35.508225 26110 tablet_bootstrap.cc:492] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:35.517553 26110 tablet_bootstrap.cc:654] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:35.519090 26108 tablet_bootstrap.cc:492] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:35.519729 26108 ts_tablet_manager.cc:1397] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.193s user 0.013s sys 0.009s
W20260807 11:09:35.522370 25723 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 11:09:35.523474 26108 raft_consensus.cc:348] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.524217 26108 raft_consensus.cc:374] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:35.526369 26108 raft_consensus.cc:729] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:35.527814 26108 consensus_queue.cc:260] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.530089 26109 tablet_bootstrap.cc:654] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:35.532509 26110 tablet_bootstrap.cc:492] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:35.532997 26110 ts_tablet_manager.cc:1397] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.005s
I20260807 11:09:35.534638 26108 ts_tablet_manager.cc:1428] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.015s user 0.003s sys 0.007s
I20260807 11:09:35.535704 26108 tablet_bootstrap.cc:492] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:35.549149 26110 raft_consensus.cc:348] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.549875 26110 raft_consensus.cc:374] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:35.550136 26110 raft_consensus.cc:729] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:35.550889 26110 consensus_queue.cc:260] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.553114 26110 ts_tablet_manager.cc:1428] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260807 11:09:35.557706 26110 tablet_bootstrap.cc:492] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:35.554323 26108 tablet_bootstrap.cc:654] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:35.562052 26109 tablet_bootstrap.cc:492] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:35.568113 26109 ts_tablet_manager.cc:1397] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.085s user 0.021s sys 0.018s
I20260807 11:09:35.562152 26390 raft_consensus.cc:482] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:35.569655 26390 raft_consensus.cc:504] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.571587 26109 raft_consensus.cc:348] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.572327 26109 raft_consensus.cc:374] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:35.572613 26109 raft_consensus.cc:729] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:35.573434 26109 consensus_queue.cc:260] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.576923 26109 ts_tablet_manager.cc:1428] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.008s user 0.006s sys 0.003s
I20260807 11:09:35.588048 26390 leader_election.cc:288] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:35.590679 26110 tablet_bootstrap.cc:654] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:35.592260 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:09:35.603978 25750 leader_election.cc:341] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:35.609776 26109 tablet_bootstrap.cc:492] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:35.610167 26280 consensus_queue.cc:1035] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:35.598243 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:35.611649 26037 raft_consensus.cc:2454] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 0.
I20260807 11:09:35.613423 25752 leader_election.cc:302] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:35.615489 26320 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:35.616917 26176 raft_consensus.cc:2781] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:35.617516 26176 raft_consensus.cc:482] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:35.619032 26176 raft_consensus.cc:3037] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:35.616268 26320 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.630781 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:09:35.633672 25750 leader_election.cc:341] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:35.637765 26320 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:35.640683 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:35.641489 26108 tablet_bootstrap.cc:492] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:35.642266 26108 ts_tablet_manager.cc:1397] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.107s user 0.024s sys 0.025s
I20260807 11:09:35.641959 26037 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 0.
I20260807 11:09:35.644308 25752 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:35.650038 26109 tablet_bootstrap.cc:654] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:35.655197 26390 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:35.655174 26108 raft_consensus.cc:348] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.656209 26108 raft_consensus.cc:374] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:35.656600 26108 raft_consensus.cc:729] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:35.656075 26390 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.662576 26280 consensus_queue.cc:1035] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:35.657383 26108 consensus_queue.cc:260] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.683130 26302 raft_consensus.cc:482] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:35.684180 26108 ts_tablet_manager.cc:1428] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.042s user 0.007s sys 0.000s
I20260807 11:09:35.683128 26110 tablet_bootstrap.cc:492] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:35.652076 26320 raft_consensus.cc:2781] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:35.684043 26302 raft_consensus.cc:504] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.685762 26110 ts_tablet_manager.cc:1397] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.129s user 0.029s sys 0.031s
I20260807 11:09:35.686210 26108 tablet_bootstrap.cc:492] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:35.688027 26320 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:35.692624 26320 raft_consensus.cc:3037] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:35.705291 26109 tablet_bootstrap.cc:492] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:35.705930 26109 ts_tablet_manager.cc:1397] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.121s user 0.009s sys 0.018s
I20260807 11:09:35.713840 26033 raft_consensus.cc:3037] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:35.725409 26109 raft_consensus.cc:348] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.726847 26109 raft_consensus.cc:374] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:35.727432 26109 raft_consensus.cc:729] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:35.728469 26109 consensus_queue.cc:260] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.735816 26176 raft_consensus.cc:504] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:35.744987 25872 consensus_peers.cc:487] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc -> Peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Couldn't send request to peer b22d69db45194908ad3d345eba9badd9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:35.740350 26176 leader_election.cc:288] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:35.766788 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:35.768042 26037 raft_consensus.cc:3037] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:35.771585 26108 tablet_bootstrap.cc:654] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:35.757951 26176 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:35.756634 26302 leader_election.cc:288] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:35.773535 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:35.783286 26176 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.786556 26390 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:35.792193 26320 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.795847 26429 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:35.794783 26176 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:35.801363 26429 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.804601 26280 consensus_queue.cc:1035] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:35.805884 26429 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:35.807713 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:35.809047 26039 raft_consensus.cc:2379] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 1: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:09:35.810184 26280 raft_consensus.cc:482] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:35.811108 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:35.810907 26280 raft_consensus.cc:504] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.814761 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:35.815523 26110 raft_consensus.cc:348] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.816335 26110 raft_consensus.cc:374] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:35.816638 26110 raft_consensus.cc:729] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:35.817763 26320 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:35.818404 26110 consensus_queue.cc:260] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.818462 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:35.821878 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:35.833205 26110 ts_tablet_manager.cc:1428] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.144s user 0.004s sys 0.004s
I20260807 11:09:35.834633 26110 tablet_bootstrap.cc:492] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:35.847836 25920 raft_consensus.cc:3037] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:35.858870 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:35.873768 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:35.874235 26046 raft_consensus.cc:3037] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
W20260807 11:09:35.875881 25752 leader_election.cc:341] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:35.877162 26461 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:35.877887 26461 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.879889 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:35.892095 26456 consensus_queue.cc:1035] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:35.899919 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:35.902808 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:35.991483 26426 raft_consensus.cc:482] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:35.998080 26426 raft_consensus.cc:504] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:35.999933 26037 raft_consensus.cc:2454] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:09:35.991557 26378 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:36.002991 26378 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.011689 26280 leader_election.cc:288] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:09:36.011883 25750 leader_election.cc:341] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:36.022328 25750 leader_election.cc:302] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:36.043197 26109 ts_tablet_manager.cc:1428] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.337s user 0.012s sys 0.024s
I20260807 11:09:36.074345 26109 tablet_bootstrap.cc:492] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:36.089389 26110 tablet_bootstrap.cc:654] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:36.087401 26378 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:36.093019 26460 raft_consensus.cc:2726] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:36.093673 26108 tablet_bootstrap.cc:492] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:36.093918 26458 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:36.094105 26108 ts_tablet_manager.cc:1397] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.408s user 0.052s sys 0.056s
I20260807 11:09:36.094374 26458 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.098076 26458 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:36.102226 26108 raft_consensus.cc:348] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.102926 26108 raft_consensus.cc:374] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:36.103195 26108 raft_consensus.cc:729] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:36.098114 25919 raft_consensus.cc:3037] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:36.103803 26108 consensus_queue.cc:260] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.104146 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:36.107048 26280 raft_consensus.cc:482] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:36.106827 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:36.107172 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:36.107702 25808 raft_consensus.cc:2454] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 0.
I20260807 11:09:36.107904 25806 raft_consensus.cc:2454] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 0.
I20260807 11:09:36.095186 25752 leader_election.cc:302] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:36.108985 26280 raft_consensus.cc:504] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.110224 26302 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:36.106748 26108 ts_tablet_manager.cc:1428] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260807 11:09:36.112084 26280 leader_election.cc:288] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:36.112223 26463 raft_consensus.cc:2781] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:36.112442 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:36.112742 26463 raft_consensus.cc:686] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:09:36.113117 26037 raft_consensus.cc:2454] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 0.
I20260807 11:09:36.110644 26302 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.085035 25869 leader_election.cc:302] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:36.090256 26461 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:09:36.087272 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 75452.2us
I20260807 11:09:36.114933 26302 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:09:36.120352 25750 leader_election.cc:341] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
W20260807 11:09:36.110702 25752 leader_election.cc:341] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:36.116475 26426 leader_election.cc:288] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:36.113631 26463 consensus_queue.cc:237] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.087801 26046 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:09:36.123449 26467 raft_consensus.cc:2726] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:36.125839 26109 tablet_bootstrap.cc:654] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
W20260807 11:09:36.122015 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 43783.5us
I20260807 11:09:36.136247 26108 tablet_bootstrap.cc:492] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
W20260807 11:09:36.139007 25750 leader_election.cc:341] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:36.139534 25750 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:36.140241 26458 raft_consensus.cc:2726] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:36.154875 25752 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:36.157416 26458 raft_consensus.cc:2781] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:36.160431 26458 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:36.160058 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:36.160780 26458 raft_consensus.cc:3037] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
W20260807 11:09:36.164057 25750 leader_election.cc:341] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
W20260807 11:09:36.149516 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 48813.8us
I20260807 11:09:36.169648 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:09:36.156704 25981 leader_election.cc:341] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:36.178133 25981 leader_election.cc:302] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:09:36.169551 26320 consensus_queue.cc:1035] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:36.176079 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:36.181043 26461 raft_consensus.cc:2726] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:36.182027 25752 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:36.163125 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:36.178741 26280 consensus_queue.cc:1035] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:36.186097 25926 raft_consensus.cc:3037] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:36.188983 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:36.184346 26320 raft_consensus.cc:2781] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:36.191769 25981 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:36.192094 26320 raft_consensus.cc:686] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:09:36.194665 26320 consensus_queue.cc:237] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.198973 26426 raft_consensus.cc:2781] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:36.199602 26426 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:36.200148 26426 raft_consensus.cc:3037] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:36.219254 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:36.204711 25981 leader_election.cc:302] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:09:36.241528 25750 leader_election.cc:341] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:36.237509 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:36.261857 26041 raft_consensus.cc:2454] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
I20260807 11:09:36.237990 26461 raft_consensus.cc:2781] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:36.263651 26461 raft_consensus.cc:482] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:36.264345 26461 raft_consensus.cc:3037] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
W20260807 11:09:36.243412 25869 leader_election.cc:341] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:36.307147 26110 tablet_bootstrap.cc:492] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:36.307688 26110 ts_tablet_manager.cc:1397] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.474s user 0.034s sys 0.036s
I20260807 11:09:36.312863 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:36.319067 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:09:36.319787 25872 consensus_peers.cc:487] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc -> Peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Couldn't send request to peer b22d69db45194908ad3d345eba9badd9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260807 11:09:36.296253 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 47104.4us
I20260807 11:09:36.321300 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:36.326004 25872 leader_election.cc:302] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:09:36.328547 26429 raft_consensus.cc:2781] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:36.329150 26429 raft_consensus.cc:482] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:36.329484 26429 raft_consensus.cc:3037] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:36.325238 26110 raft_consensus.cc:348] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.333395 26110 raft_consensus.cc:374] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:36.337028 26110 raft_consensus.cc:729] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:36.337754 26110 consensus_queue.cc:260] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.341650 26110 ts_tablet_manager.cc:1428] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.034s user 0.006s sys 0.001s
I20260807 11:09:36.342901 26110 tablet_bootstrap.cc:492] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:36.361927 26109 tablet_bootstrap.cc:492] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:36.362524 26109 ts_tablet_manager.cc:1397] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.295s user 0.022s sys 0.024s
I20260807 11:09:36.332077 25981 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:09:36.373066 25660 catalog_manager.cc:5495] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 reported cstate change: term changed from 0 to 1, leader changed from <none> to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:36.375252 25660 catalog_manager.cc:5495] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 reported cstate change: term changed from 0 to 1, leader changed from <none> to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:36.379323 26109 raft_consensus.cc:348] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.380129 26109 raft_consensus.cc:374] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:36.379961 26458 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.384464 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:36.386409 26108 tablet_bootstrap.cc:654] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:36.380558 26109 raft_consensus.cc:729] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
W20260807 11:09:36.388478 25750 leader_election.cc:341] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:36.323776 25869 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:36.391101 26109 consensus_queue.cc:260] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.395655 26458 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:36.405936 26109 ts_tablet_manager.cc:1428] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.043s user 0.006s sys 0.003s
I20260807 11:09:36.401211 26280 raft_consensus.cc:2726] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260807 11:09:36.400523 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 68577.3us
W20260807 11:09:36.416651 25981 leader_election.cc:341] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:36.402284 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:09:36.402197 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 78561.4us
I20260807 11:09:36.430120 25869 leader_election.cc:302] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:36.432803 26109 tablet_bootstrap.cc:492] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:36.437749 26477 raft_consensus.cc:2726] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:36.441833 26110 tablet_bootstrap.cc:654] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:36.397995 26488 raft_consensus.cc:2726] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:36.458065 26461 raft_consensus.cc:504] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.456267 26426 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.470299 26426 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:36.470978 26039 raft_consensus.cc:2379] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 1: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:09:36.474090 25752 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:36.474560 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:36.475826 26458 raft_consensus.cc:2726] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260807 11:09:36.476572 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
W20260807 11:09:36.477437 25981 leader_election.cc:341] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:36.477898 25808 raft_consensus.cc:2379] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 1: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:09:36.479584 25981 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:09:36.481523 26488 raft_consensus.cc:2726] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260807 11:09:36.484555 26039 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:36.466725 26429 raft_consensus.cc:504] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.499265 26108 tablet_bootstrap.cc:492] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:36.499872 26108 ts_tablet_manager.cc:1397] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.364s user 0.016s sys 0.018s
I20260807 11:09:36.509155 26461 leader_election.cc:288] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:36.511724 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:36.512745 25799 raft_consensus.cc:3037] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:36.514796 26429 leader_election.cc:288] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:36.517467 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:36.523757 26041 raft_consensus.cc:3037] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:36.523900 26108 raft_consensus.cc:348] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.519518 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:36.512741 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:36.526840 26110 tablet_bootstrap.cc:492] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
W20260807 11:09:36.526903 25750 consensus_peers.cc:487] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 -> Peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Couldn't send request to peer 4fd6381544574ddca559ce63b9243efc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:36.530401 26109 tablet_bootstrap.cc:654] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:36.530903 26108 raft_consensus.cc:374] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:36.531605 26110 ts_tablet_manager.cc:1397] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.189s user 0.017s sys 0.024s
I20260807 11:09:36.531865 26108 raft_consensus.cc:729] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:36.531548 25808 raft_consensus.cc:3037] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:36.533052 26108 consensus_queue.cc:260] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.538166 26108 ts_tablet_manager.cc:1428] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.038s user 0.004s sys 0.005s
W20260807 11:09:36.541028 25981 leader_election.cc:341] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:36.543924 26108 tablet_bootstrap.cc:492] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:36.546200 26320 consensus_queue.cc:1035] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:36.549104 26110 raft_consensus.cc:348] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.550066 26110 raft_consensus.cc:374] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:36.550375 26110 raft_consensus.cc:729] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:36.555887 26110 consensus_queue.cc:260] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.573089 26110 ts_tablet_manager.cc:1428] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.041s user 0.009s sys 0.000s
I20260807 11:09:36.576190 26110 tablet_bootstrap.cc:492] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:36.578439 26108 tablet_bootstrap.cc:654] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:36.617916 25799 raft_consensus.cc:2454] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:09:36.620608 25981 leader_election.cc:302] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:36.621778 26488 raft_consensus.cc:2781] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:36.622478 26488 raft_consensus.cc:686] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:09:36.623052 26110 tablet_bootstrap.cc:654] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
W20260807 11:09:36.627599 25750 consensus_peers.cc:487] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 -> Peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Couldn't send request to peer 4fd6381544574ddca559ce63b9243efc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:36.624040 26488 consensus_queue.cc:237] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.590309 26041 raft_consensus.cc:2454] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:09:36.632035 25872 leader_election.cc:302] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:36.633730 26477 raft_consensus.cc:2781] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:36.634366 26477 raft_consensus.cc:686] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:09:36.635478 26477 consensus_queue.cc:237] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.590799 25808 raft_consensus.cc:2454] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:09:36.669202 26109 tablet_bootstrap.cc:492] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:36.670567 26109 ts_tablet_manager.cc:1397] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.238s user 0.019s sys 0.021s
I20260807 11:09:36.675444 26109 raft_consensus.cc:348] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.676239 26109 raft_consensus.cc:374] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:36.676525 26109 raft_consensus.cc:729] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:36.687947 26109 consensus_queue.cc:260] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.689054 26473 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:36.689793 26473 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.696344 26110 tablet_bootstrap.cc:492] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:36.697046 26110 ts_tablet_manager.cc:1397] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.121s user 0.015s sys 0.010s
I20260807 11:09:36.697228 26109 ts_tablet_manager.cc:1428] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.026s user 0.005s sys 0.003s
I20260807 11:09:36.708488 26473 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:36.711283 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:36.714166 26109 tablet_bootstrap.cc:492] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
W20260807 11:09:36.715154 25981 leader_election.cc:341] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:36.717057 26110 raft_consensus.cc:348] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.717783 26110 raft_consensus.cc:374] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:36.718039 26110 raft_consensus.cc:729] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:36.718722 26110 consensus_queue.cc:260] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.720222 26110 ts_tablet_manager.cc:1428] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.023s user 0.003s sys 0.000s
I20260807 11:09:36.722585 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:36.725872 26110 tablet_bootstrap.cc:492] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
W20260807 11:09:36.726058 25981 leader_election.cc:341] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:36.726547 25981 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:09:36.739739 26473 raft_consensus.cc:2726] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:36.748831 26108 tablet_bootstrap.cc:492] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:36.749687 26108 ts_tablet_manager.cc:1397] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.206s user 0.011s sys 0.018s
I20260807 11:09:36.749410 26458 consensus_queue.cc:1035] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:36.756398 26108 raft_consensus.cc:348] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.757294 26108 raft_consensus.cc:374] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:36.757597 26108 raft_consensus.cc:729] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:36.758313 26108 consensus_queue.cc:260] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.778388 26108 ts_tablet_manager.cc:1428] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.028s user 0.007s sys 0.000s
I20260807 11:09:36.777804 25654 catalog_manager.cc:5495] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:36.779803 26108 tablet_bootstrap.cc:492] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:36.786893 26458 raft_consensus.cc:482] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:36.787972 26458 raft_consensus.cc:504] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.790429 26109 tablet_bootstrap.cc:654] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:36.819367 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:09:36.821432 25750 leader_election.cc:341] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:36.822726 26110 tablet_bootstrap.cc:654] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:36.829945 26477 raft_consensus.cc:482] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:36.830832 26108 tablet_bootstrap.cc:654] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
W20260807 11:09:36.823040 25723 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 11:09:36.823927 26458 leader_election.cc:288] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:36.826469 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:09:36.835125 25752 leader_election.cc:341] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:36.831037 26477 raft_consensus.cc:504] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.835923 25752 leader_election.cc:302] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:36.841735 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:36.842481 26460 raft_consensus.cc:2726] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:36.845544 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:36.848142 25872 leader_election.cc:302] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:36.848948 26378 raft_consensus.cc:2726] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:36.845577 26477 leader_election.cc:288] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:36.865800 25659 catalog_manager.cc:5495] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260807 11:09:36.869329 26110 tablet_bootstrap.cc:492] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:36.869848 26110 ts_tablet_manager.cc:1397] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.144s user 0.014s sys 0.007s
I20260807 11:09:36.874740 26110 raft_consensus.cc:348] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.875730 26110 raft_consensus.cc:374] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:36.876070 26110 raft_consensus.cc:729] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:36.879866 26110 consensus_queue.cc:260] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.884600 26110 ts_tablet_manager.cc:1428] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.014s user 0.007s sys 0.002s
I20260807 11:09:36.892053 26110 tablet_bootstrap.cc:492] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:36.894027 26488 raft_consensus.cc:482] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:36.894508 26488 raft_consensus.cc:504] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.896447 26108 tablet_bootstrap.cc:492] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:36.896936 26108 ts_tablet_manager.cc:1397] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.117s user 0.012s sys 0.011s
I20260807 11:09:36.899300 26488 leader_election.cc:288] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:36.901176 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:36.904223 26460 raft_consensus.cc:482] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:36.905221 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:36.904798 26460 raft_consensus.cc:504] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.907770 25981 leader_election.cc:302] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:09:36.908932 26460 leader_election.cc:288] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:36.908995 26488 raft_consensus.cc:2726] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:36.909191 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:09:36.910810 25750 leader_election.cc:341] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:36.910877 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:09:36.914520 25752 leader_election.cc:341] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:36.915158 25752 leader_election.cc:302] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:36.917886 26109 tablet_bootstrap.cc:492] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:36.919783 26108 raft_consensus.cc:348] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.920498 26108 raft_consensus.cc:374] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:36.920897 26108 raft_consensus.cc:729] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:36.920445 25921 raft_consensus.cc:3037] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:36.923457 26460 raft_consensus.cc:2726] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:36.927466 26110 tablet_bootstrap.cc:654] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:36.921623 26108 consensus_queue.cc:260] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.933868 26108 ts_tablet_manager.cc:1428] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.037s user 0.007s sys 0.000s
I20260807 11:09:36.935190 26108 tablet_bootstrap.cc:492] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:36.918373 26109 ts_tablet_manager.cc:1397] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.217s user 0.035s sys 0.014s
I20260807 11:09:36.945680 26109 raft_consensus.cc:348] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.946448 26109 raft_consensus.cc:374] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:36.946729 26109 raft_consensus.cc:729] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:36.947379 26109 consensus_queue.cc:260] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.951414 26108 tablet_bootstrap.cc:654] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:36.958561 26109 ts_tablet_manager.cc:1428] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20260807 11:09:36.960554 26109 tablet_bootstrap.cc:492] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:36.969548 26110 tablet_bootstrap.cc:492] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:36.970167 26110 ts_tablet_manager.cc:1397] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.079s user 0.012s sys 0.011s
I20260807 11:09:36.980412 26109 tablet_bootstrap.cc:654] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:36.982915 26110 raft_consensus.cc:348] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.983757 26110 raft_consensus.cc:374] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:36.984045 26110 raft_consensus.cc:729] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:36.986964 26108 tablet_bootstrap.cc:492] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:36.986521 26110 consensus_queue.cc:260] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:36.987730 26108 ts_tablet_manager.cc:1397] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.053s user 0.010s sys 0.009s
I20260807 11:09:36.999972 26110 ts_tablet_manager.cc:1428] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.029s user 0.007s sys 0.004s
I20260807 11:09:37.001166 26110 tablet_bootstrap.cc:492] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:37.019131 26108 raft_consensus.cc:348] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.020349 26108 raft_consensus.cc:374] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:37.020577 26108 raft_consensus.cc:729] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:37.024724 26108 consensus_queue.cc:260] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.025665 26109 tablet_bootstrap.cc:492] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:37.026098 26109 ts_tablet_manager.cc:1397] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.066s user 0.011s sys 0.016s
I20260807 11:09:37.027781 26108 ts_tablet_manager.cc:1428] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.040s user 0.002s sys 0.007s
I20260807 11:09:37.028227 26041 raft_consensus.cc:3037] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:37.028856 26108 tablet_bootstrap.cc:492] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:37.032719 26109 raft_consensus.cc:348] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.033511 26109 raft_consensus.cc:374] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:37.033825 26109 raft_consensus.cc:729] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:37.034480 26109 consensus_queue.cc:260] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.035161 26110 tablet_bootstrap.cc:654] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:37.042094 26109 ts_tablet_manager.cc:1428] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.015s user 0.003s sys 0.001s
W20260807 11:09:37.043928 25981 consensus_peers.cc:487] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 -> Peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Couldn't send request to peer 4fd6381544574ddca559ce63b9243efc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:37.045593 26108 tablet_bootstrap.cc:654] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:37.049530 26109 tablet_bootstrap.cc:492] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:37.053186 26429 consensus_queue.cc:1035] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:37.060148 26110 tablet_bootstrap.cc:492] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:37.060593 26110 ts_tablet_manager.cc:1397] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.060s user 0.011s sys 0.004s
I20260807 11:09:37.064267 26110 raft_consensus.cc:348] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.065043 26110 raft_consensus.cc:374] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:37.065328 26110 raft_consensus.cc:729] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:37.077234 26109 tablet_bootstrap.cc:654] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:37.070631 26110 consensus_queue.cc:260] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.082484 26110 ts_tablet_manager.cc:1428] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20260807 11:09:37.074169 26488 consensus_queue.cc:1035] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260807 11:09:37.092362 26110 tablet_bootstrap.cc:492] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:37.106769 26429 consensus_queue.cc:1035] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:37.143607 26108 tablet_bootstrap.cc:492] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:37.144251 26108 ts_tablet_manager.cc:1397] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.116s user 0.011s sys 0.015s
I20260807 11:09:37.168107 25921 raft_consensus.cc:3037] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:37.184284 26110 tablet_bootstrap.cc:654] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:37.190694 26109 tablet_bootstrap.cc:492] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:37.191179 26109 ts_tablet_manager.cc:1397] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.142s user 0.008s sys 0.020s
I20260807 11:09:37.193392 26108 raft_consensus.cc:348] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.194108 26108 raft_consensus.cc:374] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:37.194386 26108 raft_consensus.cc:729] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:37.197681 26108 consensus_queue.cc:260] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.204813 26109 raft_consensus.cc:348] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.205521 26109 raft_consensus.cc:374] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:37.205766 26109 raft_consensus.cc:729] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:37.220103 26110 tablet_bootstrap.cc:492] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:37.220606 26110 ts_tablet_manager.cc:1397] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.128s user 0.010s sys 0.009s
I20260807 11:09:37.227813 26110 raft_consensus.cc:348] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.228576 26110 raft_consensus.cc:374] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:37.228879 26110 raft_consensus.cc:729] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:37.229709 26110 consensus_queue.cc:260] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.231884 26110 ts_tablet_manager.cc:1428] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20260807 11:09:37.232820 26110 tablet_bootstrap.cc:492] T 2f60eb5499d64c89b971928b13bcd1be P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:37.200373 26108 ts_tablet_manager.cc:1428] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.056s user 0.008s sys 0.000s
I20260807 11:09:37.239756 26108 tablet_bootstrap.cc:492] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:37.240988 26426 raft_consensus.cc:482] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:37.241494 26426 raft_consensus.cc:504] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.206403 26109 consensus_queue.cc:260] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.243264 26426 leader_election.cc:288] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:37.243597 26109 ts_tablet_manager.cc:1428] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.052s user 0.005s sys 0.000s
I20260807 11:09:37.244535 26109 tablet_bootstrap.cc:492] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:37.244887 26461 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:37.244944 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:37.245409 26461 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:37.246560 25981 leader_election.cc:341] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:37.247310 26461 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:37.248947 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:37.250228 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:37.250839 26108 tablet_bootstrap.cc:654] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:37.250964 25797 raft_consensus.cc:2454] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 0.
I20260807 11:09:37.254393 26110 tablet_bootstrap.cc:654] T 2f60eb5499d64c89b971928b13bcd1be P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:37.256556 25981 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:37.257136 26109 tablet_bootstrap.cc:654] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
W20260807 11:09:37.257424 25981 leader_election.cc:341] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:37.259588 26461 raft_consensus.cc:2781] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:37.259959 26461 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:37.260259 26461 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:37.262959 26110 tablet_bootstrap.cc:492] T 2f60eb5499d64c89b971928b13bcd1be P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:37.263406 26110 ts_tablet_manager.cc:1397] T 2f60eb5499d64c89b971928b13bcd1be P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.000s
I20260807 11:09:37.265924 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:37.266486 26110 raft_consensus.cc:348] T 2f60eb5499d64c89b971928b13bcd1be P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.267576 26110 raft_consensus.cc:374] T 2f60eb5499d64c89b971928b13bcd1be P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:37.267871 26110 raft_consensus.cc:729] T 2f60eb5499d64c89b971928b13bcd1be P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:37.268719 26110 consensus_queue.cc:260] T 2f60eb5499d64c89b971928b13bcd1be P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:37.269551 25981 leader_election.cc:341] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:37.270081 25981 leader_election.cc:302] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:09:37.271310 26426 raft_consensus.cc:2726] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:37.274529 26110 ts_tablet_manager.cc:1428] T 2f60eb5499d64c89b971928b13bcd1be P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.011s user 0.005s sys 0.005s
I20260807 11:09:37.275720 26110 tablet_bootstrap.cc:492] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:37.279549 26109 tablet_bootstrap.cc:492] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:37.279989 26109 ts_tablet_manager.cc:1397] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.036s user 0.016s sys 0.000s
I20260807 11:09:37.283308 26461 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.288753 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:37.289167 26543 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:37.289816 26458 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:37.290277 26543 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.291769 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:37.292584 25797 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:37.293284 26108 tablet_bootstrap.cc:492] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:37.293830 26108 ts_tablet_manager.cc:1397] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.054s user 0.010s sys 0.000s
W20260807 11:09:37.296521 25981 leader_election.cc:341] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:37.297022 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:37.298147 25806 raft_consensus.cc:2379] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 1: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:09:37.290966 26458 raft_consensus.cc:504] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.300282 26543 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:37.302304 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:37.303642 26039 raft_consensus.cc:2379] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 1: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:09:37.303758 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:37.304823 25921 raft_consensus.cc:2454] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 0.
I20260807 11:09:37.305487 25872 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:37.306420 25750 leader_election.cc:302] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:09:37.307562 26463 raft_consensus.cc:2781] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:37.308034 26463 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:37.308596 26463 raft_consensus.cc:3037] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:37.309585 26458 leader_election.cc:288] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:37.310827 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:37.312261 26109 raft_consensus.cc:348] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:37.312822 25752 leader_election.cc:341] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:37.316219 26109 raft_consensus.cc:374] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:37.316612 26109 raft_consensus.cc:729] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:37.317119 26543 raft_consensus.cc:3037] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:37.317905 26109 consensus_queue.cc:260] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.291174 26461 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:37.321005 26109 ts_tablet_manager.cc:1428] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.041s user 0.008s sys 0.002s
I20260807 11:09:37.321913 26460 raft_consensus.cc:482] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:37.322419 26109 tablet_bootstrap.cc:492] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:37.322687 26460 raft_consensus.cc:504] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.323154 26108 raft_consensus.cc:348] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.327519 26108 raft_consensus.cc:374] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:37.327829 26108 raft_consensus.cc:729] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:37.331396 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:37.328687 26108 consensus_queue.cc:260] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:37.336211 25750 leader_election.cc:341] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:37.339187 26460 leader_election.cc:288] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:37.344471 26108 ts_tablet_manager.cc:1428] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.050s user 0.007s sys 0.001s
I20260807 11:09:37.315773 26110 tablet_bootstrap.cc:654] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:37.326297 25797 raft_consensus.cc:2454] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:09:37.355155 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:09:37.359827 25752 leader_election.cc:341] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:37.360502 25752 leader_election.cc:302] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:37.362752 26108 tablet_bootstrap.cc:492] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:37.364535 26460 raft_consensus.cc:2726] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:37.368636 26109 tablet_bootstrap.cc:654] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:37.346902 26463 raft_consensus.cc:504] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.376050 25981 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:37.377506 26461 raft_consensus.cc:2781] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:37.379793 26461 raft_consensus.cc:686] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:09:37.382838 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:37.391034 25921 raft_consensus.cc:3037] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:37.392241 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:09:37.394105 25752 leader_election.cc:341] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:37.394407 26108 tablet_bootstrap.cc:654] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:37.391762 26463 leader_election.cc:288] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:37.383489 26461 consensus_queue.cc:237] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.399523 26110 tablet_bootstrap.cc:492] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:37.399812 26429 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:37.401064 26110 ts_tablet_manager.cc:1397] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.126s user 0.013s sys 0.013s
I20260807 11:09:37.401005 26429 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.400209 26543 raft_consensus.cc:2726] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:37.402595 26109 tablet_bootstrap.cc:492] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:37.403165 26109 ts_tablet_manager.cc:1397] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.081s user 0.007s sys 0.017s
I20260807 11:09:37.404369 26110 raft_consensus.cc:348] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.406967 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:37.405367 26109 raft_consensus.cc:348] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.405404 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:37.407330 26110 raft_consensus.cc:374] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:37.407593 25797 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
I20260807 11:09:37.407904 26039 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
I20260807 11:09:37.407917 26110 raft_consensus.cc:729] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:37.406699 26429 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:37.408902 25869 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:09:37.409343 26109 raft_consensus.cc:374] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:37.409751 26109 raft_consensus.cc:729] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:37.407984 25921 raft_consensus.cc:2454] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:09:37.410071 26429 raft_consensus.cc:2781] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:37.410692 26429 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:37.410460 26109 consensus_queue.cc:260] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.411144 26429 raft_consensus.cc:3037] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:37.412027 26108 tablet_bootstrap.cc:492] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:37.412470 26108 ts_tablet_manager.cc:1397] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.058s user 0.019s sys 0.003s
I20260807 11:09:37.413015 26109 ts_tablet_manager.cc:1428] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260807 11:09:37.414175 26109 tablet_bootstrap.cc:492] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:37.415772 26108 raft_consensus.cc:348] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.416440 26108 raft_consensus.cc:374] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:37.416929 26108 raft_consensus.cc:729] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:37.417145 25750 leader_election.cc:302] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:09:37.417639 26108 consensus_queue.cc:260] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.418037 26463 raft_consensus.cc:2781] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:37.409389 26110 consensus_queue.cc:260] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.418516 26463 raft_consensus.cc:686] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:09:37.419175 26108 ts_tablet_manager.cc:1428] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260807 11:09:37.429100 26108 tablet_bootstrap.cc:492] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:37.428809 26463 consensus_queue.cc:237] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.431787 25654 catalog_manager.cc:5495] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260807 11:09:37.434943 26108 tablet_bootstrap.cc:654] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:37.436707 26110 ts_tablet_manager.cc:1428] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.035s user 0.005s sys 0.000s
I20260807 11:09:37.439808 26110 tablet_bootstrap.cc:492] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:37.440173 26429 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.445277 26109 tablet_bootstrap.cc:654] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:37.465924 26458 raft_consensus.cc:482] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:37.466550 26458 raft_consensus.cc:504] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.529048 25659 catalog_manager.cc:5495] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 reported cstate change: term changed from 0 to 1, leader changed from <none> to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:37.585804 26458 leader_election.cc:288] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:37.587708 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:37.589534 26461 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:37.590919 26461 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.595873 26461 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:37.596495 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:37.597158 26039 raft_consensus.cc:3037] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:37.597746 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:37.599450 26108 tablet_bootstrap.cc:492] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:37.599932 26108 ts_tablet_manager.cc:1397] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.171s user 0.042s sys 0.056s
I20260807 11:09:37.601481 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:37.602221 25797 raft_consensus.cc:2454] T 2f60eb5499d64c89b971928b13bcd1be P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 0.
W20260807 11:09:37.603955 25981 leader_election.cc:341] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:37.605957 25981 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:37.609251 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:37.609941 25797 raft_consensus.cc:3037] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:37.610412 26302 raft_consensus.cc:2781] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:37.598945 26429 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:37.609311 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:09:37.618741 25752 leader_election.cc:341] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:37.615774 26302 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 11:09:37.623868 25750 leader_election.cc:341] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:37.624366 25750 leader_election.cc:302] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:37.624213 26302 raft_consensus.cc:3037] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:37.640024 26039 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:09:37.642059 25797 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:09:37.642802 26426 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:37.647521 26302 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.648000 26426 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.651772 26426 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:37.632501 26564 raft_consensus.cc:2726] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:37.653010 25872 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:37.668221 26108 raft_consensus.cc:348] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.669128 26108 raft_consensus.cc:374] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:37.669438 26108 raft_consensus.cc:729] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:37.670048 26108 consensus_queue.cc:260] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.652882 26302 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:37.673518 26530 raft_consensus.cc:2781] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:37.674144 26530 raft_consensus.cc:686] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:09:37.675362 26530 consensus_queue.cc:237] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.679456 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:37.681231 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:09:37.682894 25981 leader_election.cc:341] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
W20260807 11:09:37.683751 25981 leader_election.cc:341] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:37.684100 26110 tablet_bootstrap.cc:654] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:37.685355 26109 tablet_bootstrap.cc:492] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:37.685904 26109 ts_tablet_manager.cc:1397] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.272s user 0.031s sys 0.058s
I20260807 11:09:37.681712 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:37.656738 26426 raft_consensus.cc:482] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:37.691263 25797 raft_consensus.cc:3037] T 2f60eb5499d64c89b971928b13bcd1be P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:37.702353 26108 ts_tablet_manager.cc:1428] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.102s user 0.012s sys 0.005s
I20260807 11:09:37.702214 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:37.703723 26109 raft_consensus.cc:348] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.713831 26109 raft_consensus.cc:374] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:37.714223 26109 raft_consensus.cc:729] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:37.716905 26108 tablet_bootstrap.cc:492] T adab2949574e4c6badfcf68e51b9c072 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:37.718077 26109 consensus_queue.cc:260] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.721413 25806 raft_consensus.cc:2454] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 0.
I20260807 11:09:37.716228 26426 raft_consensus.cc:504] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.859915 26109 ts_tablet_manager.cc:1428] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.174s user 0.012s sys 0.017s
I20260807 11:09:37.863003 26109 tablet_bootstrap.cc:492] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:37.863690 25981 leader_election.cc:302] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:37.869978 26473 raft_consensus.cc:2781] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:37.870376 26426 leader_election.cc:288] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:37.870494 26473 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:37.871078 26473 raft_consensus.cc:3037] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:37.872198 26110 tablet_bootstrap.cc:492] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:37.872664 26110 ts_tablet_manager.cc:1397] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.433s user 0.035s sys 0.010s
I20260807 11:09:37.878314 26110 raft_consensus.cc:348] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.879377 26110 raft_consensus.cc:374] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:37.879781 26110 raft_consensus.cc:729] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:37.880561 26110 consensus_queue.cc:260] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.881740 26180 raft_consensus.cc:482] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:37.886948 26180 raft_consensus.cc:504] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.889869 26180 leader_election.cc:288] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:37.897310 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:37.898139 25797 raft_consensus.cc:2454] T 2f60eb5499d64c89b971928b13bcd1be P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:09:37.909013 26110 ts_tablet_manager.cc:1428] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.036s user 0.007s sys 0.000s
W20260807 11:09:37.909180 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 158169us
I20260807 11:09:37.916486 26110 tablet_bootstrap.cc:492] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:37.915871 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:37.920604 26041 raft_consensus.cc:2454] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
I20260807 11:09:37.925783 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:37.927626 26565 raft_consensus.cc:482] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:37.928315 26565 raft_consensus.cc:504] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.926790 25872 leader_election.cc:302] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:37.931800 26574 raft_consensus.cc:2781] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:37.932391 26574 raft_consensus.cc:482] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:37.932969 26574 raft_consensus.cc:3037] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:37.937345 26565 leader_election.cc:288] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:37.943605 26108 tablet_bootstrap.cc:654] T adab2949574e4c6badfcf68e51b9c072 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:37.946764 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:37.948087 26033 raft_consensus.cc:2454] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 0.
I20260807 11:09:37.950419 25981 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
W20260807 11:09:37.953922 25981 leader_election.cc:341] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:37.952713 26426 raft_consensus.cc:2781] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:37.956288 26426 raft_consensus.cc:686] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:09:37.957680 26426 consensus_queue.cc:237] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.955258 25752 leader_election.cc:302] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:37.961292 26566 raft_consensus.cc:2781] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:37.962236 26566 raft_consensus.cc:482] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:37.962924 26566 raft_consensus.cc:3037] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:37.966459 26378 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:37.967267 26378 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:37.969333 25981 leader_election.cc:341] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:37.969741 25981 leader_election.cc:302] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
W20260807 11:09:37.971855 25723 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 11:09:37.970613 26302 raft_consensus.cc:2726] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:37.974846 26378 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:37.980691 26473 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.987627 26109 tablet_bootstrap.cc:654] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:37.991730 25659 catalog_manager.cc:5495] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:37.991207 26565 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:37.994895 26565 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:37.999074 26110 tablet_bootstrap.cc:654] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:37.996145 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:37.999804 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:38.003584 26565 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:38.007444 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:09:38.008479 25981 consensus_peers.cc:487] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 -> Peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Couldn't send request to peer 4fd6381544574ddca559ce63b9243efc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260807 11:09:38.011152 25872 leader_election.cc:341] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
W20260807 11:09:38.012395 25752 leader_election.cc:341] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:38.018023 26473 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:38.018216 26563 consensus_queue.cc:1035] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260807 11:09:38.037496 25750 leader_election.cc:341] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:38.020562 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:38.043691 26110 tablet_bootstrap.cc:492] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:38.044294 26110 ts_tablet_manager.cc:1397] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.130s user 0.016s sys 0.013s
W20260807 11:09:38.044536 25752 consensus_peers.cc:487] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 -> Peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Couldn't send request to peer b22d69db45194908ad3d345eba9badd9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:38.037025 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:38.047863 25926 raft_consensus.cc:2454] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 0.
I20260807 11:09:38.045542 25806 raft_consensus.cc:3037] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:38.051613 26574 raft_consensus.cc:504] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.053247 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:38.035825 26302 consensus_queue.cc:1035] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260807 11:09:38.055786 25797 raft_consensus.cc:2454] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
I20260807 11:09:38.061213 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:38.084510 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:38.085741 25796 raft_consensus.cc:2454] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
I20260807 11:09:38.085489 25660 catalog_manager.cc:5495] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
W20260807 11:09:38.089306 25981 leader_election.cc:341] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:38.093449 26110 raft_consensus.cc:348] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.094323 26110 raft_consensus.cc:374] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:38.094707 26110 raft_consensus.cc:729] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:38.095536 26110 consensus_queue.cc:260] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:38.107540 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 45859.8us
I20260807 11:09:38.108182 25750 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:09:38.113560 26564 raft_consensus.cc:2781] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:38.122728 26564 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:38.123135 26564 raft_consensus.cc:3037] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:38.129516 26108 tablet_bootstrap.cc:492] T adab2949574e4c6badfcf68e51b9c072 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:38.129992 26108 ts_tablet_manager.cc:1397] T adab2949574e4c6badfcf68e51b9c072 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.413s user 0.008s sys 0.034s
I20260807 11:09:38.142794 26477 raft_consensus.cc:482] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:38.143527 26477 raft_consensus.cc:504] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.144043 25806 raft_consensus.cc:2454] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:09:38.144954 26108 raft_consensus.cc:348] T adab2949574e4c6badfcf68e51b9c072 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.145841 26108 raft_consensus.cc:374] T adab2949574e4c6badfcf68e51b9c072 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:38.145641 25981 leader_election.cc:302] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:38.146703 26581 raft_consensus.cc:2781] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:38.147169 26566 raft_consensus.cc:504] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.146198 26108 raft_consensus.cc:729] T adab2949574e4c6badfcf68e51b9c072 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:38.162539 26477 leader_election.cc:288] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:38.181188 26566 leader_election.cc:288] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:38.163441 26110 ts_tablet_manager.cc:1428] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.119s user 0.012s sys 0.009s
I20260807 11:09:38.166127 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:38.182711 26110 tablet_bootstrap.cc:492] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
W20260807 11:09:38.182900 25872 leader_election.cc:341] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:38.183533 26581 raft_consensus.cc:686] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:09:38.168895 26574 leader_election.cc:288] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:38.170358 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:38.172304 26109 tablet_bootstrap.cc:492] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:38.118369 25869 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:09:38.185173 26109 ts_tablet_manager.cc:1397] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.322s user 0.017s sys 0.021s
I20260807 11:09:38.185935 26035 raft_consensus.cc:3037] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:38.185268 26108 consensus_queue.cc:260] T adab2949574e4c6badfcf68e51b9c072 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.196707 26108 ts_tablet_manager.cc:1428] T adab2949574e4c6badfcf68e51b9c072 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.066s user 0.010s sys 0.000s
I20260807 11:09:38.197338 26109 raft_consensus.cc:348] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.198057 26109 raft_consensus.cc:374] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:38.198535 26109 raft_consensus.cc:729] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:38.195472 26477 raft_consensus.cc:2781] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:38.199177 26109 consensus_queue.cc:260] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.203169 26108 tablet_bootstrap.cc:492] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:38.195115 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:38.200723 26477 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:38.198457 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:09:38.201268 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 89829.2us
I20260807 11:09:38.209180 26109 ts_tablet_manager.cc:1428] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.021s user 0.003s sys 0.003s
W20260807 11:09:38.211715 25750 leader_election.cc:341] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:38.202844 26564 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.194123 26581 consensus_queue.cc:237] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:38.320182 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 74635.1us
I20260807 11:09:38.209247 26039 raft_consensus.cc:3037] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:38.310570 26109 tablet_bootstrap.cc:492] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:38.207355 26477 raft_consensus.cc:3037] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
W20260807 11:09:38.323396 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 80004.4us
W20260807 11:09:38.310803 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 61856.1us
I20260807 11:09:38.339341 26110 tablet_bootstrap.cc:654] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:38.381507 26564 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:38.388592 26035 raft_consensus.cc:2454] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:09:38.398784 25872 leader_election.cc:302] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:38.401542 26378 raft_consensus.cc:2781] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:38.402390 26378 raft_consensus.cc:686] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:09:38.389528 26574 consensus_queue.cc:1035] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260807 11:09:38.406628 26461 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:38.403889 26378 consensus_queue.cc:237] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.407582 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:38.407439 26461 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.391647 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:38.417407 26477 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.416301 26578 raft_consensus.cc:482] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:38.419606 25931 raft_consensus.cc:2379] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 1: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:09:38.419571 26578 raft_consensus.cc:504] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.423946 26461 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:38.427731 26039 raft_consensus.cc:2454] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:09:38.433148 26477 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:38.441635 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:38.442059 25920 raft_consensus.cc:3037] T adab2949574e4c6badfcf68e51b9c072 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:38.453977 26108 tablet_bootstrap.cc:654] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
W20260807 11:09:38.455415 25872 leader_election.cc:341] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:38.457561 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:38.473745 26578 leader_election.cc:288] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:38.476683 26109 tablet_bootstrap.cc:654] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:38.466334 25921 raft_consensus.cc:2454] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
W20260807 11:09:38.483873 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 99066.9us
I20260807 11:09:38.485751 26559 raft_consensus.cc:482] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:38.486577 26559 raft_consensus.cc:504] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.488693 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
W20260807 11:09:38.489034 25752 leader_election.cc:341] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:38.489568 25808 raft_consensus.cc:3037] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:38.489718 25752 leader_election.cc:302] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:38.501098 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:09:38.502558 25752 leader_election.cc:341] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:38.504145 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:38.492683 26559 leader_election.cc:288] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:38.505328 25806 raft_consensus.cc:2454] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
I20260807 11:09:38.506551 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:38.507943 25981 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:38.507401 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:38.511650 25797 raft_consensus.cc:2454] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 0.
W20260807 11:09:38.513761 25981 leader_election.cc:341] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:38.522601 26599 raft_consensus.cc:2781] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:09:38.523310 26599 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:38.523953 26599 raft_consensus.cc:3037] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:38.525395 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:38.525687 26547 raft_consensus.cc:2781] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:38.526618 26547 raft_consensus.cc:686] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
W20260807 11:09:38.527120 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 112339us
I20260807 11:09:38.530138 25750 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:38.526372 25799 raft_consensus.cc:2454] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:09:38.539912 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:09:38.545535 25981 consensus_peers.cc:487] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 -> Peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Couldn't send request to peer 4fd6381544574ddca559ce63b9243efc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:38.546183 25808 raft_consensus.cc:2454] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
W20260807 11:09:38.550782 25750 leader_election.cc:341] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:38.551278 25750 leader_election.cc:302] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:38.528103 26547 consensus_queue.cc:237] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.536860 26566 raft_consensus.cc:2726] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260807 11:09:38.591137 25659 catalog_manager.cc:5495] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260807 11:09:38.601910 25981 leader_election.cc:302] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:38.596042 26565 raft_consensus.cc:2726] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:38.607343 26426 raft_consensus.cc:2781] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:38.608170 26426 raft_consensus.cc:482] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:38.615304 25869 leader_election.cc:302] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:09:38.618733 26429 raft_consensus.cc:2781] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:38.619334 26429 raft_consensus.cc:482] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:38.619904 26429 raft_consensus.cc:3037] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:38.635169 26110 tablet_bootstrap.cc:492] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:38.635720 26110 ts_tablet_manager.cc:1397] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.453s user 0.012s sys 0.028s
I20260807 11:09:38.604692 26602 consensus_queue.cc:1035] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
I20260807 11:09:38.626384 25660 catalog_manager.cc:5495] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:38.622961 26426 raft_consensus.cc:3037] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:38.643391 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:38.644198 25797 raft_consensus.cc:2379] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 1: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:09:38.652729 26110 raft_consensus.cc:348] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.665696 26110 raft_consensus.cc:374] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:38.665980 26110 raft_consensus.cc:729] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:38.666577 26110 consensus_queue.cc:260] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.673437 25869 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:38.679817 26606 raft_consensus.cc:2726] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260807 11:09:38.681048 26110 ts_tablet_manager.cc:1428] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.045s user 0.006s sys 0.001s
I20260807 11:09:38.671351 26429 raft_consensus.cc:504] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.687382 26429 leader_election.cc:288] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:38.687621 26180 consensus_queue.cc:1035] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:38.689134 26108 tablet_bootstrap.cc:492] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:38.689647 26108 ts_tablet_manager.cc:1397] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.488s user 0.027s sys 0.014s
I20260807 11:09:38.698112 26108 raft_consensus.cc:348] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.698629 26108 raft_consensus.cc:374] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:38.698846 26108 raft_consensus.cc:729] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:38.702530 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:38.700264 26108 consensus_queue.cc:260] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:38.703887 25872 leader_election.cc:341] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:38.701092 26599 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.713747 26108 ts_tablet_manager.cc:1428] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.023s user 0.008s sys 0.000s
I20260807 11:09:38.714897 26110 tablet_bootstrap.cc:492] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:38.715672 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:38.716538 25797 raft_consensus.cc:3037] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:38.721155 26108 tablet_bootstrap.cc:492] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:38.732426 26426 raft_consensus.cc:504] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.735010 26109 tablet_bootstrap.cc:492] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:38.736667 26109 ts_tablet_manager.cc:1397] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.523s user 0.025s sys 0.054s
I20260807 11:09:38.752312 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:38.756995 25796 raft_consensus.cc:3037] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:38.757350 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:38.758345 25803 raft_consensus.cc:3037] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:38.751808 25932 raft_consensus.cc:3037] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:38.766952 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:38.730420 26599 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:38.768651 25920 raft_consensus.cc:3037] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:38.774588 26108 tablet_bootstrap.cc:654] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:38.777272 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:09:38.780253 25981 leader_election.cc:341] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:38.776500 26426 leader_election.cc:288] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:38.794415 26109 raft_consensus.cc:348] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.795009 26109 raft_consensus.cc:374] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:38.795249 26109 raft_consensus.cc:729] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:38.795971 26110 tablet_bootstrap.cc:654] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:38.796973 26109 consensus_queue.cc:260] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.799712 25797 raft_consensus.cc:2454] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:09:38.801995 25869 leader_election.cc:302] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:09:38.803289 26574 raft_consensus.cc:2781] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:38.803943 26574 raft_consensus.cc:686] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:09:38.805369 26574 consensus_queue.cc:237] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.809573 26581 consensus_queue.cc:1035] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:38.823653 25920 raft_consensus.cc:2454] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:09:38.823850 25803 raft_consensus.cc:2454] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:09:38.826303 25981 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:38.834022 26302 raft_consensus.cc:2781] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:09:38.839619 25981 leader_election.cc:302] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:38.841487 26109 ts_tablet_manager.cc:1428] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.092s user 0.003s sys 0.005s
I20260807 11:09:38.838552 26302 raft_consensus.cc:686] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 2 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:09:38.843981 26302 consensus_queue.cc:237] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.847450 25796 raft_consensus.cc:2454] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:09:38.850502 26578 raft_consensus.cc:2781] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:38.851239 26578 raft_consensus.cc:686] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:09:38.853020 26578 consensus_queue.cc:237] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.895634 26109 tablet_bootstrap.cc:492] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:38.907708 26108 tablet_bootstrap.cc:492] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:38.909015 26108 ts_tablet_manager.cc:1397] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.188s user 0.014s sys 0.023s
I20260807 11:09:38.912467 25660 catalog_manager.cc:5495] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 reported cstate change: term changed from 0 to 1, leader changed from <none> to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:38.916039 26109 tablet_bootstrap.cc:654] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:38.919445 26599 consensus_queue.cc:1035] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:38.923019 26110 tablet_bootstrap.cc:492] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:38.923533 26110 ts_tablet_manager.cc:1397] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.212s user 0.013s sys 0.028s
I20260807 11:09:38.931133 25659 catalog_manager.cc:5495] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:38.946043 26108 raft_consensus.cc:348] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.947738 26108 raft_consensus.cc:374] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:38.948168 26108 raft_consensus.cc:729] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:38.949132 26108 consensus_queue.cc:260] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.955335 26110 raft_consensus.cc:348] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.956456 26110 raft_consensus.cc:374] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:38.956827 26110 raft_consensus.cc:729] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:38.958950 26110 consensus_queue.cc:260] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.960932 26108 ts_tablet_manager.cc:1428] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.052s user 0.009s sys 0.001s
I20260807 11:09:38.973421 26110 ts_tablet_manager.cc:1428] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.050s user 0.009s sys 0.000s
I20260807 11:09:38.982640 26108 tablet_bootstrap.cc:492] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:38.986888 26574 consensus_queue.cc:1035] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:38.986977 26547 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:38.989461 26547 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:38.995914 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:09:38.996568 25750 consensus_peers.cc:487] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 -> Peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Couldn't send request to peer 4fd6381544574ddca559ce63b9243efc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260807 11:09:38.999544 25750 leader_election.cc:341] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:39.002127 26110 tablet_bootstrap.cc:492] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:39.003469 25659 catalog_manager.cc:5495] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 0 to 2, leader changed from <none> to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 2 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
W20260807 11:09:39.011412 26626 log.cc:924] Time spent T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.057s user 0.001s sys 0.000s
I20260807 11:09:39.001497 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:09:39.022421 25752 leader_election.cc:341] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:39.023005 25752 leader_election.cc:302] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:39.024243 26566 raft_consensus.cc:2726] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:38.993888 26547 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:39.029910 26429 raft_consensus.cc:482] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:39.030506 26429 raft_consensus.cc:504] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.036826 26429 leader_election.cc:288] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:39.039589 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:39.040445 25808 raft_consensus.cc:2379] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 1: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:09:39.042793 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:09:39.045854 25723 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 11:09:39.005744 25659 catalog_manager.cc:5495] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260807 11:09:39.048692 26429 consensus_queue.cc:1035] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260807 11:09:39.049782 25872 leader_election.cc:302] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:39.055203 26574 raft_consensus.cc:3037] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:39.056681 26108 tablet_bootstrap.cc:654] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:39.058833 26110 tablet_bootstrap.cc:654] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:39.051865 26566 consensus_queue.cc:1035] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260807 11:09:39.064657 26109 tablet_bootstrap.cc:492] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:39.065204 26109 ts_tablet_manager.cc:1397] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.170s user 0.016s sys 0.014s
I20260807 11:09:39.083506 26109 raft_consensus.cc:348] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.084424 26109 raft_consensus.cc:374] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:39.094872 26574 raft_consensus.cc:2726] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:39.084720 26109 raft_consensus.cc:729] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:39.104557 26109 consensus_queue.cc:260] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.106865 26109 ts_tablet_manager.cc:1428] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.041s user 0.008s sys 0.000s
I20260807 11:09:39.123206 26108 tablet_bootstrap.cc:492] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:39.125115 26108 ts_tablet_manager.cc:1397] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.150s user 0.015s sys 0.010s
I20260807 11:09:39.129725 26108 raft_consensus.cc:348] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.130239 26108 raft_consensus.cc:374] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:39.130496 26108 raft_consensus.cc:729] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:39.131071 26108 consensus_queue.cc:260] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.132390 26108 ts_tablet_manager.cc:1428] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260807 11:09:39.124037 26109 tablet_bootstrap.cc:492] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:39.162668 26108 tablet_bootstrap.cc:492] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:39.185532 26109 tablet_bootstrap.cc:654] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:39.192191 26110 tablet_bootstrap.cc:492] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:39.192816 26110 ts_tablet_manager.cc:1397] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.206s user 0.017s sys 0.044s
I20260807 11:09:39.196609 26110 raft_consensus.cc:348] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.197474 26108 tablet_bootstrap.cc:654] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:39.197837 26110 raft_consensus.cc:374] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:39.198213 26110 raft_consensus.cc:729] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:39.199294 26110 consensus_queue.cc:260] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.202543 26110 ts_tablet_manager.cc:1428] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.009s user 0.005s sys 0.002s
I20260807 11:09:39.209158 26110 tablet_bootstrap.cc:492] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:39.232271 26110 tablet_bootstrap.cc:654] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:39.232616 26109 tablet_bootstrap.cc:492] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:39.233691 26109 ts_tablet_manager.cc:1397] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.110s user 0.012s sys 0.018s
I20260807 11:09:39.237330 26109 raft_consensus.cc:348] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.238011 26109 raft_consensus.cc:374] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:39.238279 26109 raft_consensus.cc:729] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:39.238915 26109 consensus_queue.cc:260] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.242899 26578 raft_consensus.cc:482] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:39.243312 26578 raft_consensus.cc:504] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.245877 26578 leader_election.cc:288] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:39.250033 26110 tablet_bootstrap.cc:492] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:39.249711 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:39.248201 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:39.250554 26110 ts_tablet_manager.cc:1397] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.047s user 0.015s sys 0.011s
I20260807 11:09:39.250667 25796 raft_consensus.cc:2454] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 0.
I20260807 11:09:39.251375 26578 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:39.252044 26109 ts_tablet_manager.cc:1428] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20260807 11:09:39.251791 26578 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.253154 26109 tablet_bootstrap.cc:492] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:39.253793 26578 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:39.257793 26110 raft_consensus.cc:348] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.258854 26110 raft_consensus.cc:374] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:39.259210 26110 raft_consensus.cc:729] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:39.250851 25920 raft_consensus.cc:2454] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 0.
I20260807 11:09:39.259898 26110 consensus_queue.cc:260] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.263334 26110 ts_tablet_manager.cc:1428] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
I20260807 11:09:39.264325 26110 tablet_bootstrap.cc:492] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:39.253070 25981 leader_election.cc:302] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:39.267401 26599 raft_consensus.cc:2781] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:39.267896 26599 raft_consensus.cc:482] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:39.268515 26599 raft_consensus.cc:3037] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:39.254990 26108 tablet_bootstrap.cc:492] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:39.274024 26108 ts_tablet_manager.cc:1397] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.112s user 0.017s sys 0.023s
I20260807 11:09:39.275537 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:39.276708 26110 tablet_bootstrap.cc:654] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:39.277371 25920 raft_consensus.cc:2379] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 1: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:09:39.282066 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:39.283756 26426 raft_consensus.cc:482] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:39.286757 26426 raft_consensus.cc:504] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.290642 26426 leader_election.cc:288] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:39.292418 25796 raft_consensus.cc:2379] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 1: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:09:39.278067 26108 raft_consensus.cc:348] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.293149 26108 raft_consensus.cc:374] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:39.294895 26108 raft_consensus.cc:729] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:39.294847 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:39.295935 26108 consensus_queue.cc:260] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.298424 26108 ts_tablet_manager.cc:1428] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20260807 11:09:39.296875 25796 raft_consensus.cc:2379] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 1: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:09:39.300040 25981 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:09:39.301811 26302 raft_consensus.cc:3037] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:39.304198 26108 tablet_bootstrap.cc:492] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:39.304111 26547 raft_consensus.cc:482] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:39.305135 26547 raft_consensus.cc:504] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.309455 26109 tablet_bootstrap.cc:654] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:39.313438 26599 raft_consensus.cc:504] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.316799 26599 leader_election.cc:288] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:39.319895 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:39.323315 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:39.328372 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:39.331229 26041 raft_consensus.cc:3037] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:39.336011 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:39.337065 26035 raft_consensus.cc:2454] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 0.
I20260807 11:09:39.339022 26565 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:39.339463 26565 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.342346 26108 tablet_bootstrap.cc:654] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:39.331851 26547 leader_election.cc:288] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:39.333207 25932 raft_consensus.cc:3037] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:39.342198 25752 leader_election.cc:302] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:39.340131 26543 consensus_queue.cc:1035] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
W20260807 11:09:39.360149 25750 leader_election.cc:341] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:39.360150 26599 consensus_queue.cc:1035] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:39.362164 26426 consensus_queue.cc:1035] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:09:39.357121 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:39.363416 25797 raft_consensus.cc:3037] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:39.371265 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:39.371893 26566 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:39.359576 26547 raft_consensus.cc:2781] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:39.373037 26547 raft_consensus.cc:482] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:39.373381 26547 raft_consensus.cc:3037] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:39.372375 26566 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.379101 25981 leader_election.cc:302] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:09:39.381361 26566 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:39.384410 26565 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:39.384342 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:09:39.387048 25752 leader_election.cc:341] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:39.388438 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:09:39.390265 25750 leader_election.cc:341] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:39.391052 25932 raft_consensus.cc:2454] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:09:39.397293 26302 raft_consensus.cc:2726] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260807 11:09:39.397899 25752 leader_election.cc:341] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:39.398463 25752 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:39.399542 26565 raft_consensus.cc:2726] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:39.407023 25981 leader_election.cc:302] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:39.408018 26302 raft_consensus.cc:2781] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:39.408466 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:39.409574 25932 raft_consensus.cc:2454] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 0.
I20260807 11:09:39.412750 26302 raft_consensus.cc:686] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:09:39.415448 25750 leader_election.cc:302] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:09:39.416469 26565 raft_consensus.cc:2781] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:39.417349 26565 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:39.418293 26565 raft_consensus.cc:3037] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:39.420433 26302 consensus_queue.cc:237] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.444046 25797 raft_consensus.cc:2454] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:09:39.412959 26547 raft_consensus.cc:504] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.460062 26547 leader_election.cc:288] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:39.464934 26426 raft_consensus.cc:2726] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:39.586913 26110 tablet_bootstrap.cc:492] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:39.587462 26110 ts_tablet_manager.cc:1397] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.323s user 0.017s sys 0.034s
I20260807 11:09:39.590243 26109 tablet_bootstrap.cc:492] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:39.590740 26109 ts_tablet_manager.cc:1397] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.338s user 0.104s sys 0.119s
I20260807 11:09:39.591912 26110 raft_consensus.cc:348] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.592833 26110 raft_consensus.cc:374] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:39.593123 26110 raft_consensus.cc:729] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:39.593287 26109 raft_consensus.cc:348] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.594003 26109 raft_consensus.cc:374] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:39.593751 26110 consensus_queue.cc:260] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.594282 26109 raft_consensus.cc:729] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:39.594944 26109 consensus_queue.cc:260] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.595512 26110 ts_tablet_manager.cc:1428] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260807 11:09:39.597353 26109 ts_tablet_manager.cc:1428] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260807 11:09:39.597702 26110 tablet_bootstrap.cc:492] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:39.598387 26109 tablet_bootstrap.cc:492] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:39.601867 25932 raft_consensus.cc:1261] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260807 11:09:39.604425 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:39.604674 26109 tablet_bootstrap.cc:654] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:39.605033 26041 raft_consensus.cc:3037] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:39.606312 26578 consensus_queue.cc:1035] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:09:39.612323 26041 raft_consensus.cc:2454] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:09:39.620113 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:39.620970 26109 tablet_bootstrap.cc:492] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:39.621438 26109 ts_tablet_manager.cc:1397] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.000s
I20260807 11:09:39.624294 26109 raft_consensus.cc:348] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.625113 26109 raft_consensus.cc:374] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:39.625391 26109 raft_consensus.cc:729] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:39.625734 25803 raft_consensus.cc:1261] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260807 11:09:39.626032 26109 consensus_queue.cc:260] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.628719 26109 ts_tablet_manager.cc:1428] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260807 11:09:39.629745 26109 tablet_bootstrap.cc:492] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:39.635387 26109 tablet_bootstrap.cc:654] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
W20260807 11:09:39.647468 25750 leader_election.cc:341] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:39.649751 26108 tablet_bootstrap.cc:492] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:39.650278 26108 ts_tablet_manager.cc:1397] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.346s user 0.016s sys 0.017s
I20260807 11:09:39.654009 26110 tablet_bootstrap.cc:654] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:39.658466 25660 catalog_manager.cc:5495] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260807 11:09:39.650410 26648 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:39.650970 26543 consensus_queue.cc:1035] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.074s
W20260807 11:09:39.779891 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 131385us
I20260807 11:09:39.839018 26108 raft_consensus.cc:348] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.844897 26108 raft_consensus.cc:374] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:39.846632 26108 raft_consensus.cc:729] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:39.848706 26108 consensus_queue.cc:260] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:39.857313 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 96684.6us
I20260807 11:09:39.678017 26426 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:39.863413 25752 leader_election.cc:302] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
W20260807 11:09:39.864048 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 129219us
I20260807 11:09:39.801661 26565 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:39.939355 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 75987.3us
I20260807 11:09:39.951216 26108 ts_tablet_manager.cc:1428] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.301s user 0.006s sys 0.010s
I20260807 11:09:39.855813 26648 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:39.873904 26426 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:39.993672 26658 log.cc:924] Time spent T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.129s user 0.000s sys 0.001s
I20260807 11:09:40.000429 26565 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:40.005158 26108 tablet_bootstrap.cc:492] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
W20260807 11:09:40.005895 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 140756us
I20260807 11:09:39.979740 26666 raft_consensus.cc:2781] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:40.030222 26648 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:40.030750 26666 raft_consensus.cc:686] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:09:40.037578 26109 tablet_bootstrap.cc:492] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:40.038055 26109 ts_tablet_manager.cc:1397] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.409s user 0.043s sys 0.036s
I20260807 11:09:40.031605 26666 consensus_queue.cc:237] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.039881 26426 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:40.045387 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:09:40.078277 25750 leader_election.cc:341] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
W20260807 11:09:40.100281 25723 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 11:09:40.125952 26109 raft_consensus.cc:348] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.126753 26109 raft_consensus.cc:374] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:40.127038 26109 raft_consensus.cc:729] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:40.127827 26109 consensus_queue.cc:260] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.108465 26599 consensus_queue.cc:1035] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.027s
I20260807 11:09:40.138239 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:40.148700 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:40.149626 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:40.149780 25799 raft_consensus.cc:2454] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 0.
I20260807 11:09:40.150468 25926 raft_consensus.cc:3037] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:40.166116 26109 ts_tablet_manager.cc:1428] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.128s user 0.005s sys 0.005s
I20260807 11:09:40.173851 26037 raft_consensus.cc:3037] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:40.156368 26648 raft_consensus.cc:482] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:40.177816 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:40.178529 26039 raft_consensus.cc:3037] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:40.180418 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:40.181278 26648 raft_consensus.cc:504] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.183249 26648 leader_election.cc:288] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:40.184033 26108 tablet_bootstrap.cc:654] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:40.201920 26109 tablet_bootstrap.cc:492] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
W20260807 11:09:40.224848 25752 leader_election.cc:341] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:40.225400 25752 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:40.233840 26110 tablet_bootstrap.cc:492] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:40.234529 26110 ts_tablet_manager.cc:1397] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.637s user 0.013s sys 0.017s
I20260807 11:09:40.211351 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:40.250114 26039 raft_consensus.cc:2454] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
W20260807 11:09:40.253526 25750 leader_election.cc:341] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:40.253535 25752 leader_election.cc:302] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:09:40.253539 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 97843.2us
I20260807 11:09:40.250341 26565 raft_consensus.cc:2726] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:40.253598 25920 raft_consensus.cc:3037] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:40.261801 25981 leader_election.cc:302] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:40.255688 26680 raft_consensus.cc:2781] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:40.256863 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:40.276367 26680 raft_consensus.cc:686] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:09:40.277727 26680 consensus_queue.cc:237] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:40.281796 25752 leader_election.cc:341] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:40.282078 25752 leader_election.cc:302] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:40.287621 26547 raft_consensus.cc:2726] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:40.285503 25926 raft_consensus.cc:2454] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:09:40.299345 26110 raft_consensus.cc:348] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:40.299674 25981 leader_election.cc:341] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:40.301928 26110 raft_consensus.cc:374] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:40.302467 26110 raft_consensus.cc:729] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:40.303503 26110 consensus_queue.cc:260] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.293398 26599 raft_consensus.cc:2781] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:40.313434 26599 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:40.314211 26599 raft_consensus.cc:3037] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:40.349037 26699 consensus_queue.cc:1035] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260807 11:09:40.357650 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 377824us
I20260807 11:09:40.360365 26110 ts_tablet_manager.cc:1428] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.125s user 0.007s sys 0.005s
I20260807 11:09:40.367044 26110 tablet_bootstrap.cc:492] T 0fc5b1dfff1a4556abdd562afbebd768 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:40.372509 26696 consensus_queue.cc:1035] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260807 11:09:40.396724 26108 tablet_bootstrap.cc:492] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:40.397306 26108 ts_tablet_manager.cc:1397] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.392s user 0.014s sys 0.022s
I20260807 11:09:40.436673 26109 tablet_bootstrap.cc:654] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:40.444802 26108 raft_consensus.cc:348] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.445899 26108 raft_consensus.cc:374] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:40.446269 26108 raft_consensus.cc:729] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:40.449782 26110 tablet_bootstrap.cc:654] T 0fc5b1dfff1a4556abdd562afbebd768 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:40.448565 26108 consensus_queue.cc:260] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.459997 26108 ts_tablet_manager.cc:1428] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.062s user 0.008s sys 0.000s
I20260807 11:09:40.461279 26108 tablet_bootstrap.cc:492] T b1765495856a47f09ddbb303f76c635f P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:40.460332 26599 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.502452 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:09:40.507543 25981 leader_election.cc:341] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:40.499056 26599 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:40.513306 26108 tablet_bootstrap.cc:654] T b1765495856a47f09ddbb303f76c635f P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:40.520064 26302 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:09:40.526957 26302 raft_consensus.cc:504] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.548249 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:40.549266 25803 raft_consensus.cc:3037] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:40.570514 26109 tablet_bootstrap.cc:492] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:40.571133 26109 ts_tablet_manager.cc:1397] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.369s user 0.012s sys 0.012s
I20260807 11:09:40.575603 26302 leader_election.cc:288] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:09:40.583504 25981 consensus_peers.cc:487] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 -> Peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Couldn't send request to peer 4fd6381544574ddca559ce63b9243efc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260807 11:09:40.584956 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:40.586858 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:40.595948 26109 raft_consensus.cc:348] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.596377 26680 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:40.602327 25981 leader_election.cc:302] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:09:40.602577 26680 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.599164 25803 raft_consensus.cc:2454] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:09:40.600476 26109 raft_consensus.cc:374] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:40.621340 26110 tablet_bootstrap.cc:492] T 0fc5b1dfff1a4556abdd562afbebd768 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:40.632025 26109 raft_consensus.cc:729] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:40.632674 26110 ts_tablet_manager.cc:1397] T 0fc5b1dfff1a4556abdd562afbebd768 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.266s user 0.029s sys 0.022s
I20260807 11:09:40.633244 26109 consensus_queue.cc:260] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.645129 25654 catalog_manager.cc:5495] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 reported cstate change: term changed from 0 to 1, leader changed from <none> to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:40.610209 26302 raft_consensus.cc:2726] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:09:40.647233 25654 catalog_manager.cc:5495] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 reported cstate change: term changed from 0 to 1, leader changed from <none> to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:40.618157 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:40.649441 26033 raft_consensus.cc:2454] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 0.
I20260807 11:09:40.661425 25752 leader_election.cc:302] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:40.602320 26648 consensus_queue.cc:1035] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.055s
I20260807 11:09:40.632436 26680 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:40.664513 26687 raft_consensus.cc:2781] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:40.673715 26687 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:40.674321 26687 raft_consensus.cc:3037] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:40.668303 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:09:40.679617 25750 leader_election.cc:341] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:40.681830 25981 leader_election.cc:302] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:40.683122 26302 raft_consensus.cc:2781] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:40.683825 26302 raft_consensus.cc:686] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:09:40.685464 26302 consensus_queue.cc:237] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.703859 26108 tablet_bootstrap.cc:492] T b1765495856a47f09ddbb303f76c635f P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:40.704380 26108 ts_tablet_manager.cc:1397] T b1765495856a47f09ddbb303f76c635f P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.243s user 0.016s sys 0.005s
I20260807 11:09:40.705853 26109 ts_tablet_manager.cc:1428] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.134s user 0.010s sys 0.014s
I20260807 11:09:40.715896 26109 tablet_bootstrap.cc:492] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:40.741856 26108 raft_consensus.cc:348] T b1765495856a47f09ddbb303f76c635f P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.742719 26108 raft_consensus.cc:374] T b1765495856a47f09ddbb303f76c635f P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:40.743006 26108 raft_consensus.cc:729] T b1765495856a47f09ddbb303f76c635f P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:40.744007 26108 consensus_queue.cc:260] T b1765495856a47f09ddbb303f76c635f P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.741832 26110 raft_consensus.cc:348] T 0fc5b1dfff1a4556abdd562afbebd768 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.745175 26110 raft_consensus.cc:374] T 0fc5b1dfff1a4556abdd562afbebd768 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:40.745456 26110 raft_consensus.cc:729] T 0fc5b1dfff1a4556abdd562afbebd768 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:40.746024 26110 consensus_queue.cc:260] T 0fc5b1dfff1a4556abdd562afbebd768 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:40.746397 25750 consensus_peers.cc:487] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 -> Peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Couldn't send request to peer 4fd6381544574ddca559ce63b9243efc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:40.749571 26110 ts_tablet_manager.cc:1428] T 0fc5b1dfff1a4556abdd562afbebd768 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.116s user 0.003s sys 0.004s
I20260807 11:09:40.752437 26110 tablet_bootstrap.cc:492] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:40.761863 26108 ts_tablet_manager.cc:1428] T b1765495856a47f09ddbb303f76c635f P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.057s user 0.005s sys 0.005s
I20260807 11:09:40.769697 26108 tablet_bootstrap.cc:492] T d8012276091e4b63b062cb7152208a16 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:40.767874 26687 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.785475 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:40.786270 26034 raft_consensus.cc:3037] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:40.787062 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:09:40.791544 25750 leader_election.cc:341] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:40.784888 26687 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:40.804075 26110 tablet_bootstrap.cc:654] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:40.804728 26109 tablet_bootstrap.cc:654] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:40.811319 26034 raft_consensus.cc:2454] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:09:40.821558 25752 leader_election.cc:302] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:40.828946 26687 raft_consensus.cc:2781] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:40.829952 26687 raft_consensus.cc:686] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:09:40.837464 26687 consensus_queue.cc:237] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.832967 26648 consensus_queue.cc:1035] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260807 11:09:40.851709 26109 tablet_bootstrap.cc:492] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:40.852265 26109 ts_tablet_manager.cc:1397] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.137s user 0.014s sys 0.002s
I20260807 11:09:40.855954 26108 tablet_bootstrap.cc:654] T d8012276091e4b63b062cb7152208a16 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:40.855860 26109 raft_consensus.cc:348] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.877985 26109 raft_consensus.cc:374] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:40.878438 26109 raft_consensus.cc:729] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:40.865582 26110 tablet_bootstrap.cc:492] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:40.864840 25659 catalog_manager.cc:5495] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260807 11:09:40.881548 26110 ts_tablet_manager.cc:1397] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.130s user 0.015s sys 0.007s
I20260807 11:09:40.887544 26109 consensus_queue.cc:260] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.891052 26109 ts_tablet_manager.cc:1428] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.038s user 0.004s sys 0.003s
I20260807 11:09:40.892958 26109 tablet_bootstrap.cc:492] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:40.901064 26180 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:40.901731 26180 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.906961 26180 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:40.909186 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:40.911665 26036 raft_consensus.cc:2454] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:09:40.916393 25872 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:40.917397 26180 raft_consensus.cc:2781] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:09:40.917816 26180 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:40.918226 26180 raft_consensus.cc:3037] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:40.915611 26110 raft_consensus.cc:348] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.920153 26110 raft_consensus.cc:374] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:40.920545 26110 raft_consensus.cc:729] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:40.921329 26110 consensus_queue.cc:260] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.923688 26110 ts_tablet_manager.cc:1428] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.042s user 0.006s sys 0.000s
I20260807 11:09:40.924685 26110 tablet_bootstrap.cc:492] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:40.927237 25660 catalog_manager.cc:5495] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 reported cstate change: term changed from 0 to 1, leader changed from <none> to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:40.931892 26108 tablet_bootstrap.cc:492] T d8012276091e4b63b062cb7152208a16 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:40.932525 26108 ts_tablet_manager.cc:1397] T d8012276091e4b63b062cb7152208a16 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.163s user 0.012s sys 0.020s
I20260807 11:09:40.936558 26108 raft_consensus.cc:348] T d8012276091e4b63b062cb7152208a16 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.937332 26108 raft_consensus.cc:374] T d8012276091e4b63b062cb7152208a16 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:40.938148 26108 raft_consensus.cc:729] T d8012276091e4b63b062cb7152208a16 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:40.938705 26378 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:40.939389 26378 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.941331 26108 consensus_queue.cc:260] T d8012276091e4b63b062cb7152208a16 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.937783 26180 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:40.932392 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:40.948989 26378 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:40.949450 25803 raft_consensus.cc:2454] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:09:40.949893 26687 consensus_queue.cc:1035] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:40.953692 26108 ts_tablet_manager.cc:1428] T d8012276091e4b63b062cb7152208a16 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.021s user 0.007s sys 0.002s
I20260807 11:09:40.954661 26108 tablet_bootstrap.cc:492] T 437f2acbd46c43b2834bf00f08d22f65 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:40.955355 26180 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:40.960891 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:40.963135 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:40.963953 25806 raft_consensus.cc:3037] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:40.964164 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:40.965119 25800 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
W20260807 11:09:40.968152 25872 leader_election.cc:341] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:40.988901 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:40.989758 26033 raft_consensus.cc:3037] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:09:40.981159 26599 raft_consensus.cc:482] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:40.996512 25869 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:09:40.997339 26180 raft_consensus.cc:2781] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:40.997704 26180 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:40.998026 26180 raft_consensus.cc:3037] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:41.003201 26109 tablet_bootstrap.cc:654] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:41.000496 26599 raft_consensus.cc:504] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.013468 26180 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.014338 26378 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:41.018743 26180 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:41.021319 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:41.021984 25800 raft_consensus.cc:2454] T 0fc5b1dfff1a4556abdd562afbebd768 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 0.
I20260807 11:09:41.023191 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:41.023762 25801 raft_consensus.cc:3037] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:41.025215 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:41.026021 25981 leader_election.cc:302] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:41.027153 26696 raft_consensus.cc:2781] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:41.028661 26110 tablet_bootstrap.cc:654] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
W20260807 11:09:41.028005 25872 leader_election.cc:341] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:41.027504 26696 raft_consensus.cc:482] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:41.029471 26696 raft_consensus.cc:3037] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:41.029901 26599 leader_election.cc:288] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:41.016801 26378 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.034680 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:41.035737 25798 raft_consensus.cc:2379] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 1: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:09:41.037786 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:41.042214 26696 raft_consensus.cc:504] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.043386 25872 leader_election.cc:302] T b1765495856a47f09ddbb303f76c635f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:41.045503 26673 raft_consensus.cc:3037] T b1765495856a47f09ddbb303f76c635f P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:41.045910 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:41.046489 25803 raft_consensus.cc:3037] T 0fc5b1dfff1a4556abdd562afbebd768 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:41.046823 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:09:41.050278 25981 leader_election.cc:341] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:41.030689 26108 tablet_bootstrap.cc:654] T 437f2acbd46c43b2834bf00f08d22f65 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:41.032490 26378 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:41.064666 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:41.065901 25801 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:09:41.067451 25869 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:09:41.044307 25806 raft_consensus.cc:2454] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:09:41.077625 25869 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:09:41.062718 26696 leader_election.cc:288] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:09:41.083521 25981 leader_election.cc:341] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:41.082661 26683 raft_consensus.cc:2781] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:09:41.084640 26683 raft_consensus.cc:686] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:09:41.073866 26378 raft_consensus.cc:2781] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:41.085485 26683 consensus_queue.cc:237] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.085974 26378 raft_consensus.cc:686] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:09:41.086978 26378 consensus_queue.cc:237] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.088265 26687 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:41.088831 26687 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.090378 25803 raft_consensus.cc:2454] T 0fc5b1dfff1a4556abdd562afbebd768 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:09:41.062386 26033 raft_consensus.cc:2454] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:09:41.094105 25981 leader_election.cc:302] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:41.100244 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:09:41.104355 25752 leader_election.cc:341] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:41.093123 26687 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:41.109642 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:41.110190 26696 raft_consensus.cc:2781] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:41.110872 26696 raft_consensus.cc:686] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:09:41.114166 26109 tablet_bootstrap.cc:492] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:41.114810 26109 ts_tablet_manager.cc:1397] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.223s user 0.013s sys 0.008s
I20260807 11:09:41.087908 26673 raft_consensus.cc:2726] T b1765495856a47f09ddbb303f76c635f P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:41.114169 26696 consensus_queue.cc:237] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.127199 26109 raft_consensus.cc:348] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.127879 26109 raft_consensus.cc:374] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:41.128131 26109 raft_consensus.cc:729] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:41.144541 26109 consensus_queue.cc:260] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.153311 26109 ts_tablet_manager.cc:1428] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.038s user 0.008s sys 0.000s
I20260807 11:09:41.154189 26109 tablet_bootstrap.cc:492] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:41.158965 26110 tablet_bootstrap.cc:492] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:41.159538 26110 ts_tablet_manager.cc:1397] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.235s user 0.014s sys 0.019s
I20260807 11:09:41.167759 26108 tablet_bootstrap.cc:492] T 437f2acbd46c43b2834bf00f08d22f65 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:41.168318 26108 ts_tablet_manager.cc:1397] T 437f2acbd46c43b2834bf00f08d22f65 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.214s user 0.008s sys 0.012s
I20260807 11:09:41.168697 26687 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:41.169668 26687 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.172122 26687 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:41.173914 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:09:41.183156 25752 leader_election.cc:341] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
W20260807 11:09:41.188416 25723 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 11:09:41.190750 26110 raft_consensus.cc:348] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.191421 26110 raft_consensus.cc:374] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:41.191663 26110 raft_consensus.cc:729] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:41.192337 26110 consensus_queue.cc:260] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.194533 26110 ts_tablet_manager.cc:1428] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.035s user 0.004s sys 0.004s
I20260807 11:09:41.195883 26109 tablet_bootstrap.cc:654] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:41.199102 26110 tablet_bootstrap.cc:492] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:41.201529 26108 raft_consensus.cc:348] T 437f2acbd46c43b2834bf00f08d22f65 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.208233 26108 raft_consensus.cc:374] T 437f2acbd46c43b2834bf00f08d22f65 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260807 11:09:41.207793 25750 leader_election.cc:341] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:41.207583 26705 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:41.218241 26705 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.208729 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:41.210494 25750 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:41.211117 26108 raft_consensus.cc:729] T 437f2acbd46c43b2834bf00f08d22f65 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:41.207334 25932 raft_consensus.cc:3037] T d8012276091e4b63b062cb7152208a16 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:41.228473 26108 consensus_queue.cc:260] T 437f2acbd46c43b2834bf00f08d22f65 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.224735 25659 catalog_manager.cc:5495] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 0 to 2, leader changed from <none> to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 2 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:41.238597 25659 catalog_manager.cc:5495] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
W20260807 11:09:41.241860 25750 leader_election.cc:341] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:41.242242 25750 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:41.245455 26547 raft_consensus.cc:2726] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:41.247663 26110 tablet_bootstrap.cc:654] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:41.232685 26666 raft_consensus.cc:2726] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:41.263866 26705 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:41.273854 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:09:41.276054 25981 consensus_peers.cc:487] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 -> Peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Couldn't send request to peer 4fd6381544574ddca559ce63b9243efc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260807 11:09:41.276249 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:41.276754 26705 consensus_queue.cc:1035] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260807 11:09:41.278682 25981 leader_election.cc:341] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:41.255756 26108 ts_tablet_manager.cc:1428] T 437f2acbd46c43b2834bf00f08d22f65 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.087s user 0.006s sys 0.002s
I20260807 11:09:41.289472 26705 consensus_queue.cc:1035] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260807 11:09:41.293664 25981 leader_election.cc:341] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:41.294131 25981 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:09:41.295372 26705 raft_consensus.cc:2726] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:41.304800 26108 tablet_bootstrap.cc:492] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:41.307862 26109 tablet_bootstrap.cc:492] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:41.316056 25659 catalog_manager.cc:5495] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260807 11:09:41.332710 26109 ts_tablet_manager.cc:1397] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.179s user 0.016s sys 0.010s
W20260807 11:09:41.335042 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 93206.7us
I20260807 11:09:41.343966 26763 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:41.347179 26109 raft_consensus.cc:348] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.347815 26109 raft_consensus.cc:374] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:41.348062 26109 raft_consensus.cc:729] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:41.344663 26763 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.348863 26109 consensus_queue.cc:260] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.353375 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:41.355656 26109 ts_tablet_manager.cc:1428] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.022s user 0.006s sys 0.001s
I20260807 11:09:41.355602 26763 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:41.365211 26109 tablet_bootstrap.cc:492] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:41.365068 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:41.377493 25872 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:41.385221 26683 raft_consensus.cc:2726] T 437f2acbd46c43b2834bf00f08d22f65 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:41.386251 26666 consensus_queue.cc:1035] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:41.384586 26110 tablet_bootstrap.cc:492] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:41.388715 26110 ts_tablet_manager.cc:1397] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.190s user 0.009s sys 0.006s
I20260807 11:09:41.393031 26108 tablet_bootstrap.cc:654] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:41.400527 26110 raft_consensus.cc:348] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.401587 26110 raft_consensus.cc:374] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:41.401948 26110 raft_consensus.cc:729] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:41.416889 26110 consensus_queue.cc:260] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:41.430290 25750 consensus_peers.cc:487] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 -> Peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Couldn't send request to peer 4fd6381544574ddca559ce63b9243efc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:41.453676 26766 raft_consensus.cc:482] T d8012276091e4b63b062cb7152208a16 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:09:41.454636 26766 raft_consensus.cc:504] T d8012276091e4b63b062cb7152208a16 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.469692 26110 ts_tablet_manager.cc:1428] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.081s user 0.008s sys 0.000s
I20260807 11:09:41.485798 26110 tablet_bootstrap.cc:492] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:41.501308 26648 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:41.502283 26648 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.532716 26108 tablet_bootstrap.cc:492] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:41.533282 26108 ts_tablet_manager.cc:1397] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.229s user 0.021s sys 0.027s
W20260807 11:09:41.537700 26774 log.cc:924] Time spent T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.126s user 0.000s sys 0.001s
I20260807 11:09:41.539572 26109 tablet_bootstrap.cc:654] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:41.546662 26648 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:41.548964 26766 leader_election.cc:288] T d8012276091e4b63b062cb7152208a16 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:41.560317 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:41.567999 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:09:41.597359 25872 consensus_peers.cc:487] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc -> Peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Couldn't send request to peer b22d69db45194908ad3d345eba9badd9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:41.600903 26666 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:41.602795 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:41.603148 26666 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.577688 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:41.605271 26033 raft_consensus.cc:2454] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 0.
I20260807 11:09:41.606961 26666 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:09:41.611768 25750 leader_election.cc:341] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:41.612598 26763 consensus_queue.cc:1035] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:41.615160 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:41.619035 25869 leader_election.cc:302] T d8012276091e4b63b062cb7152208a16 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:41.622558 25752 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
W20260807 11:09:41.628564 25752 leader_election.cc:341] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:41.624662 26683 raft_consensus.cc:2726] T d8012276091e4b63b062cb7152208a16 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:09:41.620249 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:41.626843 26771 raft_consensus.cc:2781] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260807 11:09:41.649452 25750 leader_election.cc:341] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:41.650005 25750 leader_election.cc:302] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:41.651319 26687 raft_consensus.cc:2726] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:41.657927 26036 raft_consensus.cc:1261] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260807 11:09:41.660926 26108 raft_consensus.cc:348] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.662140 26108 raft_consensus.cc:374] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:41.662703 26108 raft_consensus.cc:729] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:41.663439 26108 consensus_queue.cc:260] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.664507 26763 consensus_queue.cc:1035] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:41.642107 26771 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:41.675030 26110 tablet_bootstrap.cc:654] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:41.673028 26771 raft_consensus.cc:3037] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:41.698304 26108 ts_tablet_manager.cc:1428] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.165s user 0.009s sys 0.001s
I20260807 11:09:41.699645 26108 tablet_bootstrap.cc:492] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:41.725322 26696 consensus_queue.cc:1035] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260807 11:09:41.729049 25800 raft_consensus.cc:1261] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260807 11:09:41.827689 25917 raft_consensus.cc:3037] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:41.832895 26109 tablet_bootstrap.cc:492] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:41.833393 26109 ts_tablet_manager.cc:1397] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.473s user 0.026s sys 0.026s
I20260807 11:09:41.850216 26763 consensus_queue.cc:1035] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
W20260807 11:09:41.861342 26785 log.cc:924] Time spent T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.120s user 0.000s sys 0.000s
I20260807 11:09:41.837623 26705 raft_consensus.cc:482] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:41.876952 26705 raft_consensus.cc:504] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.883431 26378 raft_consensus.cc:482] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:41.898563 26378 raft_consensus.cc:504] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.899722 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:09:41.901367 25981 leader_election.cc:341] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:41.908151 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:41.910321 25918 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
W20260807 11:09:41.911931 25981 leader_election.cc:341] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:41.912422 25981 leader_election.cc:302] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:09:41.915043 26668 raft_consensus.cc:2726] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:41.931782 26378 leader_election.cc:288] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:41.936853 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:09:41.945149 26787 log.cc:924] Time spent T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20260807 11:09:41.916507 26771 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.949662 26771 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:41.953365 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:41.954101 26036 raft_consensus.cc:3037] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:41.902649 26705 leader_election.cc:288] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:41.961313 26109 raft_consensus.cc:348] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.962065 26109 raft_consensus.cc:374] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:41.962316 26109 raft_consensus.cc:729] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:41.962894 26109 consensus_queue.cc:260] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:41.992803 26109 ts_tablet_manager.cc:1428] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.159s user 0.006s sys 0.002s
I20260807 11:09:41.992869 26302 raft_consensus.cc:482] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:41.997460 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:09:42.009810 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 103732us
I20260807 11:09:42.001858 26302 raft_consensus.cc:504] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.012465 26302 leader_election.cc:288] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:42.012552 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:41.996579 26426 consensus_queue.cc:1035] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260807 11:09:42.015080 26036 raft_consensus.cc:2454] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:09:42.019138 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:42.019462 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:42.020905 25752 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:42.021899 26687 raft_consensus.cc:2781] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:42.022179 26109 tablet_bootstrap.cc:492] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:42.023469 25806 raft_consensus.cc:2454] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 0.
W20260807 11:09:42.030429 25750 leader_election.cc:341] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
W20260807 11:09:42.031771 25981 leader_election.cc:341] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:42.032907 25981 leader_election.cc:302] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:42.028961 26687 raft_consensus.cc:686] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:09:42.034653 26687 consensus_queue.cc:237] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.044972 26110 tablet_bootstrap.cc:492] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:42.045516 26110 ts_tablet_manager.cc:1397] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.568s user 0.074s sys 0.075s
I20260807 11:09:42.039809 26302 raft_consensus.cc:2781] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:42.047711 25869 leader_election.cc:302] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:42.048518 26110 raft_consensus.cc:348] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.047894 26302 raft_consensus.cc:482] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:42.049256 26110 raft_consensus.cc:374] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:42.049576 26110 raft_consensus.cc:729] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:42.050184 26110 consensus_queue.cc:260] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.049438 26302 raft_consensus.cc:3037] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:42.064479 26108 tablet_bootstrap.cc:654] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:42.069406 26110 ts_tablet_manager.cc:1428] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260807 11:09:42.083669 26109 tablet_bootstrap.cc:654] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:42.090546 26110 tablet_bootstrap.cc:492] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:42.091845 26683 raft_consensus.cc:2726] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:09:42.118170 26302 raft_consensus.cc:504] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.136968 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:42.141077 25800 raft_consensus.cc:3037] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:42.141414 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:42.142999 26108 tablet_bootstrap.cc:492] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
W20260807 11:09:42.143322 25981 leader_election.cc:341] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:42.144150 26108 ts_tablet_manager.cc:1397] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.445s user 0.017s sys 0.028s
I20260807 11:09:42.149863 26108 raft_consensus.cc:348] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.147436 26110 tablet_bootstrap.cc:654] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:42.151392 26108 raft_consensus.cc:374] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:42.150498 25800 raft_consensus.cc:2454] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:09:42.145244 26302 leader_election.cc:288] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:42.154841 26109 tablet_bootstrap.cc:492] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:42.154783 25981 leader_election.cc:302] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:42.155480 26109 ts_tablet_manager.cc:1397] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.135s user 0.010s sys 0.005s
I20260807 11:09:42.155932 26302 raft_consensus.cc:2781] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:42.156437 26302 raft_consensus.cc:686] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:09:42.157424 26302 consensus_queue.cc:237] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.168831 26109 raft_consensus.cc:348] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.169520 26109 raft_consensus.cc:374] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:42.169790 26109 raft_consensus.cc:729] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:42.157869 25654 catalog_manager.cc:5495] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 reported cstate change: term changed from 0 to 1, leader changed from <none> to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:42.151757 26108 raft_consensus.cc:729] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:42.174666 26108 consensus_queue.cc:260] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.177163 26108 ts_tablet_manager.cc:1428] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20260807 11:09:42.178195 26108 tablet_bootstrap.cc:492] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:42.183203 26110 tablet_bootstrap.cc:492] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:42.183689 26110 ts_tablet_manager.cc:1397] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.104s user 0.012s sys 0.002s
I20260807 11:09:42.184832 26109 consensus_queue.cc:260] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.186389 26108 tablet_bootstrap.cc:654] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:42.187001 26109 ts_tablet_manager.cc:1428] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.031s user 0.003s sys 0.003s
I20260807 11:09:42.187244 26110 raft_consensus.cc:348] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.188053 26109 tablet_bootstrap.cc:492] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:42.188062 26110 raft_consensus.cc:374] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:42.188529 26110 raft_consensus.cc:729] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:42.191180 26110 consensus_queue.cc:260] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.193464 26110 ts_tablet_manager.cc:1428] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20260807 11:09:42.194329 26110 tablet_bootstrap.cc:492] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:42.206341 26110 tablet_bootstrap.cc:654] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:42.210610 26109 tablet_bootstrap.cc:654] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:42.216602 26108 tablet_bootstrap.cc:492] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:42.221899 26108 ts_tablet_manager.cc:1397] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.044s user 0.025s sys 0.001s
I20260807 11:09:42.224587 26108 raft_consensus.cc:348] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.225535 26108 raft_consensus.cc:374] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:42.225883 26108 raft_consensus.cc:729] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:42.226336 26672 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:42.226956 26672 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.232827 26110 tablet_bootstrap.cc:492] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:42.232926 26672 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:42.233348 26110 ts_tablet_manager.cc:1397] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.039s user 0.016s sys 0.000s
I20260807 11:09:42.226644 26108 consensus_queue.cc:260] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.235844 26110 raft_consensus.cc:348] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.236613 26110 raft_consensus.cc:374] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:42.236912 26110 raft_consensus.cc:729] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:42.236867 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:42.238075 26108 ts_tablet_manager.cc:1428] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
W20260807 11:09:42.238255 25981 leader_election.cc:341] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:42.236140 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:42.239014 25796 raft_consensus.cc:2454] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 0.
W20260807 11:09:42.241060 25723 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 11:09:42.241942 25981 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:42.241550 26110 consensus_queue.cc:260] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.242934 26672 raft_consensus.cc:2781] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:42.243117 26108 tablet_bootstrap.cc:492] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:42.243300 26672 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:42.243681 26672 raft_consensus.cc:3037] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:42.243815 26110 ts_tablet_manager.cc:1428] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260807 11:09:42.253773 26672 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.258436 26110 tablet_bootstrap.cc:492] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:42.258436 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:42.259377 25800 raft_consensus.cc:3037] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:42.267422 26110 tablet_bootstrap.cc:654] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:42.301957 25800 raft_consensus.cc:2454] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:09:42.303532 25981 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:42.305334 26791 raft_consensus.cc:2781] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:42.305851 26791 raft_consensus.cc:686] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:09:42.306981 26791 consensus_queue.cc:237] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.355368 26110 tablet_bootstrap.cc:492] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:42.356943 26678 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:42.358363 26678 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.366076 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:42.368041 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:09:42.369516 25750 leader_election.cc:341] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
W20260807 11:09:42.370611 25752 leader_election.cc:341] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:42.370896 25752 leader_election.cc:302] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:42.357667 26110 ts_tablet_manager.cc:1397] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.100s user 0.009s sys 0.006s
I20260807 11:09:42.373000 26790 raft_consensus.cc:2726] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:42.365535 26678 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:42.377440 26108 tablet_bootstrap.cc:654] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:42.378722 26672 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:42.382311 26109 tablet_bootstrap.cc:492] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:42.382884 26109 ts_tablet_manager.cc:1397] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.195s user 0.010s sys 0.016s
I20260807 11:09:42.385432 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:09:42.394424 25981 leader_election.cc:341] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:42.389312 26110 raft_consensus.cc:348] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.396334 26110 raft_consensus.cc:374] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:42.396667 26110 raft_consensus.cc:729] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:42.400748 26110 consensus_queue.cc:260] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.399792 26109 raft_consensus.cc:348] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.402808 26109 raft_consensus.cc:374] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:42.403097 26109 raft_consensus.cc:729] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:42.403924 26110 ts_tablet_manager.cc:1428] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.032s user 0.008s sys 0.000s
I20260807 11:09:42.405077 26110 tablet_bootstrap.cc:492] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815: Bootstrap starting.
I20260807 11:09:42.403864 26109 consensus_queue.cc:260] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.407722 26109 ts_tablet_manager.cc:1428] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.025s user 0.003s sys 0.005s
I20260807 11:09:42.413968 26108 tablet_bootstrap.cc:492] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:42.414434 26108 ts_tablet_manager.cc:1397] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.171s user 0.008s sys 0.008s
I20260807 11:09:42.417285 26110 tablet_bootstrap.cc:654] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:42.419560 26109 tablet_bootstrap.cc:492] T e4de8b347f214a69bbac09d43ad38db7 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:42.422760 26108 raft_consensus.cc:348] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.423426 26108 raft_consensus.cc:374] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:42.423668 26108 raft_consensus.cc:729] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:42.424288 26108 consensus_queue.cc:260] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.427366 26108 ts_tablet_manager.cc:1428] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260807 11:09:42.428138 25918 raft_consensus.cc:3037] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:42.428411 26108 tablet_bootstrap.cc:492] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:42.445438 26672 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:42.446218 26672 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.434530 25660 catalog_manager.cc:5495] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260807 11:09:42.450249 25660 catalog_manager.cc:5495] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260807 11:09:42.453590 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:42.454507 25800 raft_consensus.cc:2454] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 0.
I20260807 11:09:42.459291 25981 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:42.459755 26672 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:42.460515 26672 raft_consensus.cc:2781] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:42.460996 26672 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:42.461494 26672 raft_consensus.cc:3037] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:42.469732 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:42.470636 25916 raft_consensus.cc:2454] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 0.
I20260807 11:09:42.484450 26790 raft_consensus.cc:482] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:42.485183 26790 raft_consensus.cc:504] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.489104 26109 tablet_bootstrap.cc:654] T e4de8b347f214a69bbac09d43ad38db7 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:42.489888 26790 leader_election.cc:288] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:42.492128 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:42.493041 26036 raft_consensus.cc:2454] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 0.
I20260807 11:09:42.494504 25752 leader_election.cc:302] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:42.492304 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:42.498014 26790 raft_consensus.cc:2781] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260807 11:09:42.498281 25750 leader_election.cc:341] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:42.499953 26790 raft_consensus.cc:482] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:42.500977 26790 raft_consensus.cc:3037] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:42.505307 26108 tablet_bootstrap.cc:654] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:42.539486 26672 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.543970 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:42.544809 25796 raft_consensus.cc:3037] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:42.545406 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:42.546571 25916 raft_consensus.cc:3037] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:42.548915 26672 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:42.568284 26110 tablet_bootstrap.cc:492] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815: No bootstrap required, opened a new log
I20260807 11:09:42.568894 26110 ts_tablet_manager.cc:1397] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815: Time spent bootstrapping tablet: real 0.164s user 0.021s sys 0.000s
I20260807 11:09:42.553061 26790 raft_consensus.cc:504] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.574416 26109 tablet_bootstrap.cc:492] T e4de8b347f214a69bbac09d43ad38db7 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:42.574859 26109 ts_tablet_manager.cc:1397] T e4de8b347f214a69bbac09d43ad38db7 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.156s user 0.013s sys 0.007s
I20260807 11:09:42.583506 25926 raft_consensus.cc:3037] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
W20260807 11:09:42.588140 25981 consensus_peers.cc:487] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 -> Peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Couldn't send request to peer 4fd6381544574ddca559ce63b9243efc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:42.591307 26110 raft_consensus.cc:348] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.590700 26678 consensus_queue.cc:1035] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:42.590551 26771 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:42.595577 26771 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.590502 25796 raft_consensus.cc:2454] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:09:42.597697 26771 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:42.597885 26109 raft_consensus.cc:348] T e4de8b347f214a69bbac09d43ad38db7 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.598634 26109 raft_consensus.cc:374] T e4de8b347f214a69bbac09d43ad38db7 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:42.600098 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:42.600416 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:42.601399 26790 leader_election.cc:288] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:42.598912 26109 raft_consensus.cc:729] T e4de8b347f214a69bbac09d43ad38db7 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
W20260807 11:09:42.601773 25750 leader_election.cc:341] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
W20260807 11:09:42.602421 25750 leader_election.cc:341] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:42.602463 26109 consensus_queue.cc:260] T e4de8b347f214a69bbac09d43ad38db7 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.610082 25981 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:42.610544 26110 raft_consensus.cc:374] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:42.610821 26110 raft_consensus.cc:729] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Initialized, Role: FOLLOWER
I20260807 11:09:42.611660 26109 ts_tablet_manager.cc:1428] T e4de8b347f214a69bbac09d43ad38db7 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.037s user 0.005s sys 0.002s
I20260807 11:09:42.612499 26302 raft_consensus.cc:2781] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:42.613014 26302 raft_consensus.cc:686] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:09:42.611917 26110 consensus_queue.cc:260] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.613603 26302 consensus_queue.cc:237] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.621423 26109 tablet_bootstrap.cc:492] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:42.633122 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:42.633749 26038 raft_consensus.cc:3037] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:42.635299 26110 ts_tablet_manager.cc:1428] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815: Time spent starting tablet: real 0.066s user 0.005s sys 0.002s
I20260807 11:09:42.642784 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:42.643611 26037 raft_consensus.cc:2454] T e4de8b347f214a69bbac09d43ad38db7 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 0.
I20260807 11:09:42.612679 26790 consensus_queue.cc:1035] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.025s
I20260807 11:09:42.656603 26108 tablet_bootstrap.cc:492] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:42.657097 26108 ts_tablet_manager.cc:1397] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.229s user 0.025s sys 0.008s
I20260807 11:09:42.660122 25752 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:42.669648 26797 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:42.671715 26771 raft_consensus.cc:2781] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:42.669816 26822 raft_consensus.cc:482] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:42.699285 26108 raft_consensus.cc:348] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.700803 26108 raft_consensus.cc:374] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:42.707780 26108 raft_consensus.cc:729] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:42.703572 26797 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.700591 26771 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:42.715919 26771 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:42.718374 25916 raft_consensus.cc:2454] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:09:42.721390 26038 raft_consensus.cc:2454] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:09:42.703717 26822 raft_consensus.cc:504] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.708669 26108 consensus_queue.cc:260] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.714644 26797 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:42.732889 25752 leader_election.cc:302] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:42.739748 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:42.759354 26108 ts_tablet_manager.cc:1428] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.102s user 0.009s sys 0.000s
I20260807 11:09:42.760542 26108 tablet_bootstrap.cc:492] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:42.761207 25659 catalog_manager.cc:5495] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260807 11:09:42.770702 26109 tablet_bootstrap.cc:654] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
W20260807 11:09:42.775279 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 49843us
W20260807 11:09:42.784415 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 51638.3us
I20260807 11:09:42.772751 26790 raft_consensus.cc:2781] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:42.794552 26822 leader_election.cc:288] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:42.806150 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:09:42.811733 25752 leader_election.cc:341] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:42.813755 26180 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:09:42.814281 26180 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.793339 26790 raft_consensus.cc:686] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:09:42.825244 26687 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:42.833266 26687 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.835240 26687 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:42.835776 26108 tablet_bootstrap.cc:654] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:42.823879 26696 consensus_queue.cc:1035] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260807 11:09:42.839239 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:42.830673 26790 consensus_queue.cc:237] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:42.841749 25750 leader_election.cc:341] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:42.824599 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:09:42.842787 26771 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.852641 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:09:42.861083 25752 leader_election.cc:341] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:42.861640 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:42.861727 25752 leader_election.cc:302] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
W20260807 11:09:42.864519 25750 leader_election.cc:341] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:42.867676 25750 leader_election.cc:302] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:42.868860 26180 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:42.877526 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:42.869515 26771 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:42.879961 26109 tablet_bootstrap.cc:492] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:42.880465 26109 ts_tablet_manager.cc:1397] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.263s user 0.013s sys 0.004s
I20260807 11:09:42.880263 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:42.868081 26838 raft_consensus.cc:2726] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:42.880921 26046 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:42.872648 26678 raft_consensus.cc:2726] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:42.893213 26109 raft_consensus.cc:348] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.894754 26109 raft_consensus.cc:374] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:42.895238 26109 raft_consensus.cc:729] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:42.896071 26109 consensus_queue.cc:260] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.906064 25916 raft_consensus.cc:3037] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:42.898617 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:09:42.912611 25981 consensus_peers.cc:487] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 -> Peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Couldn't send request to peer 4fd6381544574ddca559ce63b9243efc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260807 11:09:42.912901 25750 leader_election.cc:341] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:42.916885 26046 raft_consensus.cc:2454] T e4de8b347f214a69bbac09d43ad38db7 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:09:42.923190 26109 ts_tablet_manager.cc:1428] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.042s user 0.008s sys 0.000s
I20260807 11:09:42.924551 25752 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:42.931535 26109 tablet_bootstrap.cc:492] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:42.934998 26828 raft_consensus.cc:2781] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:42.935617 26828 raft_consensus.cc:686] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:09:42.953774 26108 tablet_bootstrap.cc:492] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:42.954330 26108 ts_tablet_manager.cc:1397] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.194s user 0.011s sys 0.007s
I20260807 11:09:42.959842 25654 catalog_manager.cc:5495] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 reported cstate change: term changed from 0 to 1, leader changed from <none> to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:42.936823 26828 consensus_queue.cc:237] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.988372 25869 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:42.989318 26108 raft_consensus.cc:348] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:42.990065 26180 raft_consensus.cc:2726] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:09:42.990258 26108 raft_consensus.cc:374] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:42.990615 26108 raft_consensus.cc:729] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:42.991389 26108 consensus_queue.cc:260] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.017679 26108 ts_tablet_manager.cc:1428] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.063s user 0.004s sys 0.004s
I20260807 11:09:42.984627 26705 consensus_queue.cc:1035] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.033s
I20260807 11:09:43.018769 26108 tablet_bootstrap.cc:492] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:43.006916 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:43.003535 26672 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:43.026053 26672 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.028733 26672 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:43.030021 26109 tablet_bootstrap.cc:654] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:43.041474 25869 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:43.050655 26857 raft_consensus.cc:2726] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:09:43.064805 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:43.065490 25806 raft_consensus.cc:2454] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 0.
I20260807 11:09:43.072873 25981 leader_election.cc:302] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:09:43.078588 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:09:43.091102 25981 leader_election.cc:341] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:43.086215 25659 catalog_manager.cc:5495] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 reported cstate change: term changed from 0 to 1, leader changed from <none> to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:43.093253 26672 raft_consensus.cc:2781] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:43.094684 26705 consensus_queue.cc:1035] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260807 11:09:43.103663 26672 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:43.117030 26672 raft_consensus.cc:3037] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:43.136799 26108 tablet_bootstrap.cc:654] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:43.156131 26109 tablet_bootstrap.cc:492] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:43.156678 26109 ts_tablet_manager.cc:1397] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.230s user 0.014s sys 0.040s
I20260807 11:09:43.166960 26672 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.167181 26109 raft_consensus.cc:348] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.168030 26109 raft_consensus.cc:374] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:43.168365 26109 raft_consensus.cc:729] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:43.168967 26672 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:43.224351 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:43.234784 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:43.235949 25786 raft_consensus.cc:3037] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:43.237929 26860 consensus_queue.cc:1035] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260807 11:09:43.230036 26109 consensus_queue.cc:260] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.293473 26109 ts_tablet_manager.cc:1428] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.134s user 0.007s sys 0.002s
I20260807 11:09:43.296167 26108 tablet_bootstrap.cc:492] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:43.297327 26108 ts_tablet_manager.cc:1397] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.279s user 0.017s sys 0.028s
I20260807 11:09:43.298923 26792 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:43.300933 26792 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.303695 26109 tablet_bootstrap.cc:492] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:43.309094 26108 raft_consensus.cc:348] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:09:43.316360 25981 leader_election.cc:341] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:43.326423 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:43.327821 26792 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:43.328940 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:43.325402 26108 raft_consensus.cc:374] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:43.337363 25872 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:09:43.338572 26108 raft_consensus.cc:729] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:43.345471 26833 raft_consensus.cc:2726] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:09:43.350277 26108 consensus_queue.cc:260] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.348646 25786 raft_consensus.cc:2454] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:09:43.357389 26108 ts_tablet_manager.cc:1428] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.060s user 0.002s sys 0.007s
I20260807 11:09:43.371811 26108 tablet_bootstrap.cc:492] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:43.374315 25981 leader_election.cc:302] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:43.376922 26696 raft_consensus.cc:2781] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:43.378829 26696 raft_consensus.cc:686] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:09:43.379134 26109 tablet_bootstrap.cc:654] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:43.380300 26696 consensus_queue.cc:237] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.383652 26108 tablet_bootstrap.cc:654] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:43.452955 26842 consensus_queue.cc:1035] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:43.465088 26109 tablet_bootstrap.cc:492] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:43.465674 26109 ts_tablet_manager.cc:1397] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.163s user 0.012s sys 0.006s
I20260807 11:09:43.475857 25916 raft_consensus.cc:3037] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:43.468319 26109 raft_consensus.cc:348] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.486877 26109 raft_consensus.cc:374] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:43.487179 26109 raft_consensus.cc:729] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
W20260807 11:09:43.488919 25750 consensus_peers.cc:487] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 -> Peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Couldn't send request to peer 4fd6381544574ddca559ce63b9243efc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:43.494098 26109 consensus_queue.cc:260] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.497805 26790 consensus_queue.cc:1035] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:43.499648 25919 raft_consensus.cc:3037] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:43.526808 26109 ts_tablet_manager.cc:1428] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.061s user 0.006s sys 0.001s
I20260807 11:09:43.527824 26036 raft_consensus.cc:3037] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:43.528283 26109 tablet_bootstrap.cc:492] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:43.538125 26426 raft_consensus.cc:482] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:43.538671 26426 raft_consensus.cc:504] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.540709 26426 leader_election.cc:288] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:43.544093 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:09:43.546203 25723 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 11:09:43.546396 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:43.547099 25786 raft_consensus.cc:2454] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 0.
W20260807 11:09:43.545537 25981 leader_election.cc:341] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:43.549239 25981 leader_election.cc:302] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:43.549953 26860 raft_consensus.cc:2781] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:43.550284 26860 raft_consensus.cc:482] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:43.550595 26860 raft_consensus.cc:3037] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:43.553355 25654 catalog_manager.cc:5495] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260807 11:09:43.562741 26108 tablet_bootstrap.cc:492] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:43.563220 26108 ts_tablet_manager.cc:1397] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.192s user 0.016s sys 0.006s
I20260807 11:09:43.567638 26108 raft_consensus.cc:348] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.568399 26108 raft_consensus.cc:374] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:43.568702 26108 raft_consensus.cc:729] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:43.569532 26108 consensus_queue.cc:260] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.572103 26108 ts_tablet_manager.cc:1428] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260807 11:09:43.573206 26108 tablet_bootstrap.cc:492] T 6eed5249afac4e16a4b17938ad6e1353 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
W20260807 11:09:43.577275 25750 consensus_peers.cc:487] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 -> Peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Couldn't send request to peer 4fd6381544574ddca559ce63b9243efc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:43.579624 26860 raft_consensus.cc:504] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.585743 26860 leader_election.cc:288] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:43.586273 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:43.587100 25786 raft_consensus.cc:3037] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:43.598805 26109 tablet_bootstrap.cc:654] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:43.601586 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:09:43.605479 26108 tablet_bootstrap.cc:654] T 6eed5249afac4e16a4b17938ad6e1353 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
W20260807 11:09:43.613385 25981 leader_election.cc:341] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Illegal state: must be running to vote when last-logged opid is not known
I20260807 11:09:43.608480 25786 raft_consensus.cc:2454] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:09:43.629576 25981 leader_election.cc:302] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:09:43.630268 26860 raft_consensus.cc:2781] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:43.630764 26860 raft_consensus.cc:686] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:09:43.631603 26860 consensus_queue.cc:237] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.638253 26109 tablet_bootstrap.cc:492] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:43.638684 26109 ts_tablet_manager.cc:1397] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.111s user 0.009s sys 0.005s
I20260807 11:09:43.640731 26109 raft_consensus.cc:348] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.641428 26109 raft_consensus.cc:374] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:43.641618 26109 raft_consensus.cc:729] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:43.642244 26109 consensus_queue.cc:260] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.644527 26109 ts_tablet_manager.cc:1428] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.006s user 0.000s sys 0.004s
I20260807 11:09:43.645399 26109 tablet_bootstrap.cc:492] T f9465d2caa3e43c8891f52350177df8d P b22d69db45194908ad3d345eba9badd9: Bootstrap starting.
I20260807 11:09:43.651032 26109 tablet_bootstrap.cc:654] T f9465d2caa3e43c8891f52350177df8d P b22d69db45194908ad3d345eba9badd9: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:43.651502 26108 tablet_bootstrap.cc:492] T 6eed5249afac4e16a4b17938ad6e1353 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:43.651983 26108 ts_tablet_manager.cc:1397] T 6eed5249afac4e16a4b17938ad6e1353 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.079s user 0.011s sys 0.006s
I20260807 11:09:43.655912 26108 raft_consensus.cc:348] T 6eed5249afac4e16a4b17938ad6e1353 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.658869 26108 raft_consensus.cc:374] T 6eed5249afac4e16a4b17938ad6e1353 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:43.664868 26108 raft_consensus.cc:729] T 6eed5249afac4e16a4b17938ad6e1353 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:43.665666 26108 consensus_queue.cc:260] T 6eed5249afac4e16a4b17938ad6e1353 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.669996 26108 ts_tablet_manager.cc:1428] T 6eed5249afac4e16a4b17938ad6e1353 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.018s user 0.006s sys 0.001s
I20260807 11:09:43.671041 26108 tablet_bootstrap.cc:492] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:43.676029 26109 tablet_bootstrap.cc:492] T f9465d2caa3e43c8891f52350177df8d P b22d69db45194908ad3d345eba9badd9: No bootstrap required, opened a new log
I20260807 11:09:43.676471 26109 ts_tablet_manager.cc:1397] T f9465d2caa3e43c8891f52350177df8d P b22d69db45194908ad3d345eba9badd9: Time spent bootstrapping tablet: real 0.031s user 0.013s sys 0.013s
I20260807 11:09:43.677314 26108 tablet_bootstrap.cc:654] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:43.679149 26109 raft_consensus.cc:348] T f9465d2caa3e43c8891f52350177df8d P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.679982 26109 raft_consensus.cc:374] T f9465d2caa3e43c8891f52350177df8d P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:43.680239 26109 raft_consensus.cc:729] T f9465d2caa3e43c8891f52350177df8d P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Initialized, Role: FOLLOWER
I20260807 11:09:43.681017 26109 consensus_queue.cc:260] T f9465d2caa3e43c8891f52350177df8d P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.683830 26109 ts_tablet_manager.cc:1428] T f9465d2caa3e43c8891f52350177df8d P b22d69db45194908ad3d345eba9badd9: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260807 11:09:43.695156 26108 tablet_bootstrap.cc:492] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:43.695755 26108 ts_tablet_manager.cc:1397] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.003s
I20260807 11:09:43.699366 26108 raft_consensus.cc:348] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.700064 26108 raft_consensus.cc:374] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:43.700340 26108 raft_consensus.cc:729] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:43.700950 26108 consensus_queue.cc:260] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.702898 26108 ts_tablet_manager.cc:1428] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260807 11:09:43.703969 26108 tablet_bootstrap.cc:492] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:43.709508 26108 tablet_bootstrap.cc:654] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:43.732988 26180 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:43.733595 26180 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.735951 26180 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:43.746083 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:43.747021 25786 raft_consensus.cc:2454] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
I20260807 11:09:43.747781 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:43.748222 25869 leader_election.cc:302] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:09:43.748442 26036 raft_consensus.cc:2454] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
I20260807 11:09:43.753865 26108 tablet_bootstrap.cc:492] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:43.754376 26108 ts_tablet_manager.cc:1397] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.051s user 0.011s sys 0.032s
I20260807 11:09:43.756415 26180 raft_consensus.cc:2781] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:43.756989 26180 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:43.757355 26180 raft_consensus.cc:3037] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:43.758283 26108 raft_consensus.cc:348] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.758550 25654 catalog_manager.cc:5495] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260807 11:09:43.759017 26108 raft_consensus.cc:374] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:43.759359 26108 raft_consensus.cc:729] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:43.760185 26108 consensus_queue.cc:260] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.762216 26108 ts_tablet_manager.cc:1428] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260807 11:09:43.763590 26108 tablet_bootstrap.cc:492] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:43.768991 26108 tablet_bootstrap.cc:654] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:43.780041 26108 tablet_bootstrap.cc:492] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:43.780484 26108 ts_tablet_manager.cc:1397] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.010s
I20260807 11:09:43.783075 26108 raft_consensus.cc:348] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.783746 26108 raft_consensus.cc:374] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:43.783995 26108 raft_consensus.cc:729] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:43.784559 26108 consensus_queue.cc:260] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.787220 26108 ts_tablet_manager.cc:1428] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260807 11:09:43.788049 26108 tablet_bootstrap.cc:492] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc: Bootstrap starting.
I20260807 11:09:43.789155 26180 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.792881 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:43.793433 25786 raft_consensus.cc:3037] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:43.793586 26108 tablet_bootstrap.cc:654] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc: Neither blocks nor log segments found. Creating new log.
I20260807 11:09:43.802542 25786 raft_consensus.cc:2454] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:09:43.803747 25869 leader_election.cc:302] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:09:43.804721 26683 raft_consensus.cc:2781] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:43.805006 26180 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:43.805409 26683 raft_consensus.cc:686] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:09:43.806535 26683 consensus_queue.cc:237] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.810369 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:43.811146 26036 raft_consensus.cc:3037] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:43.832083 26108 tablet_bootstrap.cc:492] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc: No bootstrap required, opened a new log
I20260807 11:09:43.832604 26108 ts_tablet_manager.cc:1397] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc: Time spent bootstrapping tablet: real 0.045s user 0.024s sys 0.012s
I20260807 11:09:43.835893 26108 raft_consensus.cc:348] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.836588 26108 raft_consensus.cc:374] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:09:43.836879 26108 raft_consensus.cc:729] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Initialized, Role: FOLLOWER
I20260807 11:09:43.838687 26108 consensus_queue.cc:260] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.841180 26108 ts_tablet_manager.cc:1428] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260807 11:09:43.848984 26036 raft_consensus.cc:2454] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:09:43.857260 25660 catalog_manager.cc:5495] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:43.869102 26860 consensus_queue.cc:1035] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:09:43.902252 25916 raft_consensus.cc:3037] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:43.909847 26302 consensus_queue.cc:1035] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:09:43.911434 25919 raft_consensus.cc:3037] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:43.963294 26180 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:09:43.963816 26180 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.966502 26180 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:43.967146 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:09:43.967830 25800 raft_consensus.cc:2454] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
I20260807 11:09:43.969020 25869 leader_election.cc:302] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:09:43.972887 26792 raft_consensus.cc:2781] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:09:43.973273 26792 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:09:43.973619 26792 raft_consensus.cc:3037] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:43.975515 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:09:43.976207 26036 raft_consensus.cc:2454] T f9465d2caa3e43c8891f52350177df8d P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 0.
I20260807 11:09:43.985514 26792 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:43.989761 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:09:43.990489 25800 raft_consensus.cc:3037] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:44.003512 26792 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:44.004631 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:09:44.004844 25800 raft_consensus.cc:2454] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:09:44.005348 26036 raft_consensus.cc:3037] T f9465d2caa3e43c8891f52350177df8d P b22d69db45194908ad3d345eba9badd9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:44.006333 25869 leader_election.cc:302] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:09:44.007859 26180 raft_consensus.cc:2781] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:09:44.008296 26180 raft_consensus.cc:686] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:09:44.009061 26180 consensus_queue.cc:237] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:09:44.024650 25919 raft_consensus.cc:3037] T 6eed5249afac4e16a4b17938ad6e1353 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:44.026665 26036 raft_consensus.cc:2454] T f9465d2caa3e43c8891f52350177df8d P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:09:44.033444 26302 consensus_queue.cc:1035] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:09:44.044799 26860 consensus_queue.cc:1035] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:09:44.047636 25654 catalog_manager.cc:5495] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:44.051180 25926 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:09:44.296368 26180 consensus_queue.cc:1035] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:09:44.298218 26180 consensus_queue.cc:1035] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:09:44.423907 26683 consensus_queue.cc:1035] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:09:44.424991 26180 consensus_queue.cc:1035] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:09:46.071986 24034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:09:46.108263 26927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:09:46.109407 26924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:09:46.110157 26925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:09:46.112942 24034 server_base.cc:1034] running on GCE node
I20260807 11:09:46.115993 24034 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 11:09:46.116266 24034 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 11:09:46.116474 24034 hybrid_clock.cc:648] HybridClock initialized: now 1786100986116455 us; error 0 us; skew 500 ppm
I20260807 11:09:46.117110 24034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:09:46.124051 24034 webserver.cc:460] Webserver started at http://127.23.120.132:42031/ using document root <none> and password file <none>
I20260807 11:09:46.124784 24034 fs_manager.cc:359] Metadata directory not provided
I20260807 11:09:46.125099 24034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:09:46.125569 24034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:09:46.127287 24034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-3-root/instance:
uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
format_stamp: "Formatted at 2026-08-07 11:09:46 on dist-test-slave-g170"
I20260807 11:09:46.134423 24034 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20260807 11:09:46.138226 26935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:09:46.139335 24034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 11:09:46.139665 24034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-3-root
uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
format_stamp: "Formatted at 2026-08-07 11:09:46 on dist-test-slave-g170"
I20260807 11:09:46.140054 24034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:09:46.189672 24034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:09:46.191147 24034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:09:46.196943 24034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:09:46.201164 24034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:09:46.201503 24034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:09:46.201862 24034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:09:46.202107 24034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:09:46.337687 24034 rpc_server.cc:307] RPC server started. Bound to: 127.23.120.132:43279
I20260807 11:09:46.337792 27031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.120.132:43279 every 8 connection(s)
I20260807 11:09:46.374993 24034 auto_leader_rebalancer-test.cc:413] add a tserver: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:09:46.402072 27033 heartbeater.cc:347] Connected to a master server at 127.23.120.190:37321
I20260807 11:09:46.402643 27033 heartbeater.cc:464] Registering TS with master...
I20260807 11:09:46.403625 27033 heartbeater.cc:511] Master 127.23.120.190:37321 requested a full tablet report, sending...
I20260807 11:09:46.406131 25654 ts_manager.cc:194] Registered new tserver with Master: eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:09:46.407948 25654 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34692
I20260807 11:09:46.768894 25916 consensus_queue.cc:237] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:46.782665 26046 raft_consensus.cc:1261] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:46.784675 26923 consensus_queue.cc:1035] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:09:46.800339 25786 raft_consensus.cc:1261] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:46.802587 26833 consensus_queue.cc:1035] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260807 11:09:46.811050 25869 consensus_peers.cc:487] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5d5c9aff2844213a67d710585224083. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:46.819950 27047 raft_consensus.cc:2932] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:46.825227 25800 raft_consensus.cc:2932] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:46.833679 26036 raft_consensus.cc:2932] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:46.848382 25654 catalog_manager.cc:5495] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815 reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:46.912518 25803 consensus_queue.cc:237] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:46.926729 26036 raft_consensus.cc:1261] T e4de8b347f214a69bbac09d43ad38db7 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:46.931350 26920 consensus_queue.cc:1035] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:09:46.941558 25919 raft_consensus.cc:1261] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:46.943799 27066 consensus_queue.cc:1035] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260807 11:09:46.955053 25750 consensus_peers.cc:487] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4de8b347f214a69bbac09d43ad38db7. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:46.965098 26842 raft_consensus.cc:2932] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:46.967056 26046 raft_consensus.cc:2932] T e4de8b347f214a69bbac09d43ad38db7 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:46.987545 25926 raft_consensus.cc:2932] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:47.009656 25803 consensus_queue.cc:237] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:47.012306 25660 catalog_manager.cc:5495] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:47.022981 26046 raft_consensus.cc:1261] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:47.023761 25919 raft_consensus.cc:1261] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 11:09:47.024027 25750 consensus_peers.cc:487] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f31f65eddcdf48f6a64e0a1ddc0c6695. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:47.024287 26920 consensus_queue.cc:1035] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:47.025207 26842 consensus_queue.cc:1035] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:47.074555 27066 raft_consensus.cc:2932] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:47.082955 25919 raft_consensus.cc:2932] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:47.090019 25800 consensus_queue.cc:237] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:47.090209 26046 raft_consensus.cc:2932] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:47.096088 25659 catalog_manager.cc:5495] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260807 11:09:47.122740 25750 consensus_peers.cc:487] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99e5feeda1eb4affb2532bda1b0055cd. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:47.123646 25919 raft_consensus.cc:1261] T 99e5feeda1eb4affb2532bda1b0055cd P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:47.123646 26046 raft_consensus.cc:1261] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:47.125146 27076 consensus_queue.cc:1035] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:09:47.126648 26842 consensus_queue.cc:1035] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:47.154105 26842 raft_consensus.cc:2932] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:47.156155 26036 raft_consensus.cc:2932] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:47.164351 25926 raft_consensus.cc:2932] T 99e5feeda1eb4affb2532bda1b0055cd P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:47.178268 25654 catalog_manager.cc:5495] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:47.323146 27084 ts_tablet_manager.cc:927] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195)
I20260807 11:09:47.344398 27084 tablet_copy_client.cc:316] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.129:40195
I20260807 11:09:47.382263 27090 ts_tablet_manager.cc:927] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:47.387292 27090 tablet_copy_client.cc:316] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.130:36465
I20260807 11:09:47.409127 25825 tablet_copy_service.cc:140] P 606d49e053a346e486f48cb31832b815: Received BeginTabletCopySession request for tablet e4de8b347f214a69bbac09d43ad38db7 from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:59118)
I20260807 11:09:47.409934 25825 tablet_copy_service.cc:161] P 606d49e053a346e486f48cb31832b815: Beginning new tablet copy session on tablet e4de8b347f214a69bbac09d43ad38db7 from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:59118: session id = eb1fca62dcb34ed9b32188ccb19f0543-e4de8b347f214a69bbac09d43ad38db7
I20260807 11:09:47.417163 27033 heartbeater.cc:503] Master 127.23.120.190:37321 was elected leader, sending a full tablet report...
I20260807 11:09:47.426373 25825 tablet_copy_source_session.cc:205] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:09:47.429518 25945 tablet_copy_service.cc:140] P 4fd6381544574ddca559ce63b9243efc: Received BeginTabletCopySession request for tablet d5d5c9aff2844213a67d710585224083 from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:35276)
I20260807 11:09:47.430266 25945 tablet_copy_service.cc:161] P 4fd6381544574ddca559ce63b9243efc: Beginning new tablet copy session on tablet d5d5c9aff2844213a67d710585224083 from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:35276: session id = eb1fca62dcb34ed9b32188ccb19f0543-d5d5c9aff2844213a67d710585224083
I20260807 11:09:47.440227 27084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4de8b347f214a69bbac09d43ad38db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:47.443357 25945 tablet_copy_source_session.cc:205] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:09:47.447039 27090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5d5c9aff2844213a67d710585224083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:47.460115 27084 tablet_copy_client.cc:791] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:09:47.460978 27084 tablet_copy_client.cc:655] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:09:47.468940 27090 tablet_copy_client.cc:791] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:09:47.469883 27090 tablet_copy_client.cc:655] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:09:47.474740 27090 tablet_copy_client.cc:531] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:09:47.487320 27090 tablet_bootstrap.cc:492] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
I20260807 11:09:47.487625 27084 tablet_copy_client.cc:531] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:09:47.518890 27084 tablet_bootstrap.cc:492] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
I20260807 11:09:47.556371 27090 tablet_bootstrap.cc:492] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:09:47.557842 27090 tablet_bootstrap.cc:492] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:09:47.558691 27084 tablet_bootstrap.cc:492] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:09:47.558710 27090 ts_tablet_manager.cc:1397] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 0.072s user 0.020s sys 0.009s
I20260807 11:09:47.566771 27084 tablet_bootstrap.cc:492] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:09:47.567574 27084 ts_tablet_manager.cc:1397] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 0.071s user 0.020s sys 0.016s
I20260807 11:09:47.569582 27090 raft_consensus.cc:348] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:47.570755 27090 raft_consensus.cc:729] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:09:47.570495 27084 raft_consensus.cc:348] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:47.571506 27084 raft_consensus.cc:729] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:09:47.572321 27084 consensus_queue.cc:260] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:47.572458 27090 consensus_queue.cc:260] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:47.579783 27090 ts_tablet_manager.cc:1428] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.015s user 0.006s sys 0.004s
I20260807 11:09:47.593710 27084 ts_tablet_manager.cc:1428] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.026s user 0.012s sys 0.012s
I20260807 11:09:47.593660 25945 tablet_copy_service.cc:342] P 4fd6381544574ddca559ce63b9243efc: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-d5d5c9aff2844213a67d710585224083 received from {username='slave'} at 127.0.0.1:35276
I20260807 11:09:47.594252 25945 tablet_copy_service.cc:433] P 4fd6381544574ddca559ce63b9243efc: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-d5d5c9aff2844213a67d710585224083 on tablet d5d5c9aff2844213a67d710585224083 with peer eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:09:47.597234 25825 tablet_copy_service.cc:342] P 606d49e053a346e486f48cb31832b815: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-e4de8b347f214a69bbac09d43ad38db7 received from {username='slave'} at 127.0.0.1:59118
I20260807 11:09:47.597638 25825 tablet_copy_service.cc:433] P 606d49e053a346e486f48cb31832b815: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-e4de8b347f214a69bbac09d43ad38db7 on tablet e4de8b347f214a69bbac09d43ad38db7 with peer eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:09:47.617887 27084 ts_tablet_manager.cc:927] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195)
I20260807 11:09:47.631533 27084 tablet_copy_client.cc:316] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.129:40195
I20260807 11:09:47.633792 25825 tablet_copy_service.cc:140] P 606d49e053a346e486f48cb31832b815: Received BeginTabletCopySession request for tablet f31f65eddcdf48f6a64e0a1ddc0c6695 from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:59118)
I20260807 11:09:47.634335 25825 tablet_copy_service.cc:161] P 606d49e053a346e486f48cb31832b815: Beginning new tablet copy session on tablet f31f65eddcdf48f6a64e0a1ddc0c6695 from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:59118: session id = eb1fca62dcb34ed9b32188ccb19f0543-f31f65eddcdf48f6a64e0a1ddc0c6695
I20260807 11:09:47.641300 25825 tablet_copy_source_session.cc:205] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:09:47.645007 27084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f31f65eddcdf48f6a64e0a1ddc0c6695. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:47.661644 27084 tablet_copy_client.cc:791] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:09:47.662304 27084 tablet_copy_client.cc:655] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:09:47.666647 27084 tablet_copy_client.cc:531] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:09:47.677568 27084 tablet_bootstrap.cc:492] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
I20260807 11:09:47.711930 27084 tablet_bootstrap.cc:492] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:09:47.713339 27084 tablet_bootstrap.cc:492] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:09:47.714078 27084 ts_tablet_manager.cc:1397] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 0.037s user 0.023s sys 0.010s
I20260807 11:09:47.716955 27084 raft_consensus.cc:348] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:47.717797 27084 raft_consensus.cc:729] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:09:47.718444 27084 consensus_queue.cc:260] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:47.720821 27084 ts_tablet_manager.cc:1428] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260807 11:09:47.722613 25825 tablet_copy_service.cc:342] P 606d49e053a346e486f48cb31832b815: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-f31f65eddcdf48f6a64e0a1ddc0c6695 received from {username='slave'} at 127.0.0.1:59118
I20260807 11:09:47.723031 25825 tablet_copy_service.cc:433] P 606d49e053a346e486f48cb31832b815: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-f31f65eddcdf48f6a64e0a1ddc0c6695 on tablet f31f65eddcdf48f6a64e0a1ddc0c6695 with peer eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:09:47.730116 27090 ts_tablet_manager.cc:927] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195)
I20260807 11:09:47.732403 27090 tablet_copy_client.cc:316] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.129:40195
I20260807 11:09:47.734840 25825 tablet_copy_service.cc:140] P 606d49e053a346e486f48cb31832b815: Received BeginTabletCopySession request for tablet 99e5feeda1eb4affb2532bda1b0055cd from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:59118)
I20260807 11:09:47.735404 25825 tablet_copy_service.cc:161] P 606d49e053a346e486f48cb31832b815: Beginning new tablet copy session on tablet 99e5feeda1eb4affb2532bda1b0055cd from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:59118: session id = eb1fca62dcb34ed9b32188ccb19f0543-99e5feeda1eb4affb2532bda1b0055cd
I20260807 11:09:47.744714 25825 tablet_copy_source_session.cc:205] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:09:47.748307 27090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e5feeda1eb4affb2532bda1b0055cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:47.767904 27090 tablet_copy_client.cc:791] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:09:47.768554 27090 tablet_copy_client.cc:655] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:09:47.774704 27090 tablet_copy_client.cc:531] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:09:47.784202 27090 tablet_bootstrap.cc:492] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
I20260807 11:09:47.812680 27090 tablet_bootstrap.cc:492] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:09:47.813889 27090 tablet_bootstrap.cc:492] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:09:47.814734 27090 ts_tablet_manager.cc:1397] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 0.031s user 0.024s sys 0.004s
I20260807 11:09:47.908070 26987 raft_consensus.cc:1203] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:09:47.962132 27090 raft_consensus.cc:348] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:47.963982 27090 raft_consensus.cc:729] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:09:47.964953 27090 consensus_queue.cc:260] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:47.970808 27090 ts_tablet_manager.cc:1428] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.156s user 0.066s sys 0.034s
I20260807 11:09:47.972504 25825 tablet_copy_service.cc:342] P 606d49e053a346e486f48cb31832b815: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-99e5feeda1eb4affb2532bda1b0055cd received from {username='slave'} at 127.0.0.1:59118
I20260807 11:09:47.972980 25825 tablet_copy_service.cc:433] P 606d49e053a346e486f48cb31832b815: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-99e5feeda1eb4affb2532bda1b0055cd on tablet 99e5feeda1eb4affb2532bda1b0055cd with peer eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:09:47.979519 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 54617.2us
I20260807 11:09:47.981462 26987 raft_consensus.cc:1203] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:09:48.127490 26987 raft_consensus.cc:1203] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:09:48.193056 26987 raft_consensus.cc:1203] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:09:48.311275 27047 raft_consensus.cc:1050] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:09:48.313997 27047 consensus_queue.cc:237] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:48.328011 26987 raft_consensus.cc:1261] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:48.328070 26036 raft_consensus.cc:1261] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:48.329406 27051 consensus_queue.cc:1035] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:48.330335 27047 consensus_queue.cc:1035] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:48.330904 25800 raft_consensus.cc:1261] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:48.337777 27047 consensus_queue.cc:1035] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:48.350353 27051 raft_consensus.cc:2932] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:48.354218 26036 raft_consensus.cc:2932] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:48.356675 26987 raft_consensus.cc:2932] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:48.369778 25659 catalog_manager.cc:5495] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:48.373440 25803 raft_consensus.cc:2932] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:48.406699 25919 consensus_queue.cc:237] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:48.417472 26987 raft_consensus.cc:1261] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:48.418712 27047 consensus_queue.cc:1035] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:09:48.426028 26038 raft_consensus.cc:1261] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:48.426818 26987 raft_consensus.cc:2932] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:48.430389 27051 consensus_queue.cc:1035] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:09:48.425346 27047 raft_consensus.cc:2932] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:48.440192 26038 raft_consensus.cc:2932] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:48.454288 25659 catalog_manager.cc:5495] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:48.462630 25637 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d5d5c9aff2844213a67d710585224083 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 11:09:48.472121 25785 tablet_service.cc:1514] Processing DeleteTablet for tablet d5d5c9aff2844213a67d710585224083 with delete_type TABLET_DATA_TOMBSTONED (TS 606d49e053a346e486f48cb31832b815 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53306
I20260807 11:09:48.476639 27132 tablet_replica.cc:330] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815: stopping tablet replica
I20260807 11:09:48.477941 27132 raft_consensus.cc:2229] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:48.478729 27132 raft_consensus.cc:2258] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:48.489377 27132 ts_tablet_manager.cc:1905] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 11:09:48.505771 27132 ts_tablet_manager.cc:1918] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 11:09:48.506237 27132 log.cc:1196] T d5d5c9aff2844213a67d710585224083 P 606d49e053a346e486f48cb31832b815: Deleting WAL directory at /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-0-root/wals/d5d5c9aff2844213a67d710585224083
I20260807 11:09:48.508002 25637 catalog_manager.cc:4841] TS 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): tablet d5d5c9aff2844213a67d710585224083 (table test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]) successfully deleted
I20260807 11:09:48.521837 26842 raft_consensus.cc:1050] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:09:48.523990 26842 consensus_queue.cc:237] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:48.529295 26036 raft_consensus.cc:1261] T e4de8b347f214a69bbac09d43ad38db7 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:48.529965 25919 raft_consensus.cc:1261] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:48.530164 26987 raft_consensus.cc:1261] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:48.531286 26842 consensus_queue.cc:1035] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:48.532184 27120 consensus_queue.cc:1035] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:48.533084 27082 consensus_queue.cc:1035] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:48.552353 27082 raft_consensus.cc:2932] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:48.553866 25926 raft_consensus.cc:2932] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:48.557274 26987 raft_consensus.cc:2932] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:48.570339 26036 raft_consensus.cc:2932] T e4de8b347f214a69bbac09d43ad38db7 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:48.575603 25660 catalog_manager.cc:5495] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:09:48.624501 25800 consensus_queue.cc:237] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:48.651834 25926 raft_consensus.cc:1261] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:48.652935 26987 raft_consensus.cc:1261] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:48.656023 27082 consensus_queue.cc:1035] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:09:48.660629 26842 consensus_queue.cc:1035] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:09:48.668318 27082 raft_consensus.cc:2932] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:48.671095 25919 raft_consensus.cc:2932] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:48.678213 26986 raft_consensus.cc:2932] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:48.683463 25637 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e4de8b347f214a69bbac09d43ad38db7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 11:09:48.684008 26842 raft_consensus.cc:1050] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:09:48.687011 26842 consensus_queue.cc:237] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:48.696116 25654 catalog_manager.cc:5495] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 reported cstate change: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:09:48.712337 27117 raft_consensus.cc:1050] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:09:48.717792 26046 raft_consensus.cc:1261] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:48.715188 27117 consensus_queue.cc:237] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:48.719085 26986 raft_consensus.cc:1261] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:48.721418 25919 raft_consensus.cc:1261] T 99e5feeda1eb4affb2532bda1b0055cd P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:48.724532 26032 tablet_service.cc:1514] Processing DeleteTablet for tablet e4de8b347f214a69bbac09d43ad38db7 with delete_type TABLET_DATA_TOMBSTONED (TS b22d69db45194908ad3d345eba9badd9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59300
I20260807 11:09:48.721200 27120 consensus_queue.cc:1035] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:09:48.734818 27120 consensus_queue.cc:1035] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:09:48.746359 27082 consensus_queue.cc:1035] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:09:48.761703 26986 raft_consensus.cc:1261] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:48.764674 26036 raft_consensus.cc:1261] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:48.769229 25919 raft_consensus.cc:1261] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:48.784078 27141 tablet_replica.cc:330] T e4de8b347f214a69bbac09d43ad38db7 P b22d69db45194908ad3d345eba9badd9: stopping tablet replica
I20260807 11:09:48.785609 27141 raft_consensus.cc:2229] T e4de8b347f214a69bbac09d43ad38db7 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:48.786677 27141 raft_consensus.cc:2258] T e4de8b347f214a69bbac09d43ad38db7 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:48.801816 27139 consensus_queue.cc:1035] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260807 11:09:48.803488 27120 consensus_queue.cc:1035] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260807 11:09:48.803520 27142 log.cc:924] Time spent T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.050s user 0.001s sys 0.000s
I20260807 11:09:48.805429 27143 consensus_queue.cc:1035] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:09:48.806237 27141 ts_tablet_manager.cc:1905] T e4de8b347f214a69bbac09d43ad38db7 P b22d69db45194908ad3d345eba9badd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 11:09:48.812096 27120 raft_consensus.cc:2932] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:48.819581 25917 raft_consensus.cc:2932] T 99e5feeda1eb4affb2532bda1b0055cd P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:48.835371 26987 raft_consensus.cc:2932] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:48.858018 25926 raft_consensus.cc:2932] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:48.857026 27139 raft_consensus.cc:2932] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:48.869670 25654 catalog_manager.cc:5495] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:48.902439 26046 raft_consensus.cc:2932] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:48.916981 26038 raft_consensus.cc:2932] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:48.920197 27141 ts_tablet_manager.cc:1918] T e4de8b347f214a69bbac09d43ad38db7 P b22d69db45194908ad3d345eba9badd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 11:09:48.920907 27141 log.cc:1196] T e4de8b347f214a69bbac09d43ad38db7 P b22d69db45194908ad3d345eba9badd9: Deleting WAL directory at /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-2-root/wals/e4de8b347f214a69bbac09d43ad38db7
I20260807 11:09:48.923871 25639 catalog_manager.cc:4841] TS b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): tablet e4de8b347f214a69bbac09d43ad38db7 (table test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]) successfully deleted
I20260807 11:09:48.903602 26986 raft_consensus.cc:2932] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:48.942984 25659 catalog_manager.cc:5495] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 4fd6381544574ddca559ce63b9243efc reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:49.000481 25800 consensus_queue.cc:237] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:49.012068 26046 raft_consensus.cc:1261] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:49.015983 26986 raft_consensus.cc:1261] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:49.017180 27120 consensus_queue.cc:1035] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:09:49.018527 27143 consensus_queue.cc:1035] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:09:49.034344 27139 raft_consensus.cc:2932] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:49.035568 26046 raft_consensus.cc:2932] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:49.041318 25801 consensus_queue.cc:237] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:49.049345 26038 raft_consensus.cc:1261] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:49.051879 27117 consensus_queue.cc:1035] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:09:49.051210 26986 raft_consensus.cc:2932] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:49.062146 26987 raft_consensus.cc:1261] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:49.064812 27152 consensus_queue.cc:1035] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:09:49.066099 25654 catalog_manager.cc:5495] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 reported cstate change: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:49.068784 26842 raft_consensus.cc:2932] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:49.074931 25637 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet f31f65eddcdf48f6a64e0a1ddc0c6695 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 11:09:49.076894 26046 raft_consensus.cc:2932] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:49.086906 25637 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 99e5feeda1eb4affb2532bda1b0055cd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 11:09:49.095798 26987 raft_consensus.cc:2932] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:49.100278 25910 tablet_service.cc:1514] Processing DeleteTablet for tablet f31f65eddcdf48f6a64e0a1ddc0c6695 with delete_type TABLET_DATA_TOMBSTONED (TS 4fd6381544574ddca559ce63b9243efc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39730
I20260807 11:09:49.123507 27153 tablet_replica.cc:330] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 4fd6381544574ddca559ce63b9243efc: stopping tablet replica
I20260807 11:09:49.125365 27153 raft_consensus.cc:2229] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:49.125006 25654 catalog_manager.cc:5495] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:49.126467 27153 raft_consensus.cc:2258] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:49.130914 27153 ts_tablet_manager.cc:1905] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 4fd6381544574ddca559ce63b9243efc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 11:09:49.142083 25910 tablet_service.cc:1514] Processing DeleteTablet for tablet 99e5feeda1eb4affb2532bda1b0055cd with delete_type TABLET_DATA_TOMBSTONED (TS 4fd6381544574ddca559ce63b9243efc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39730
I20260807 11:09:49.151124 27153 ts_tablet_manager.cc:1918] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 4fd6381544574ddca559ce63b9243efc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 11:09:49.151573 27153 log.cc:1196] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 4fd6381544574ddca559ce63b9243efc: Deleting WAL directory at /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-1-root/wals/f31f65eddcdf48f6a64e0a1ddc0c6695
I20260807 11:09:49.153128 25637 catalog_manager.cc:4841] TS 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): tablet f31f65eddcdf48f6a64e0a1ddc0c6695 (table test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]) successfully deleted
I20260807 11:09:49.153472 27153 tablet_replica.cc:330] T 99e5feeda1eb4affb2532bda1b0055cd P 4fd6381544574ddca559ce63b9243efc: stopping tablet replica
I20260807 11:09:49.154646 27153 raft_consensus.cc:2229] T 99e5feeda1eb4affb2532bda1b0055cd P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:49.155565 27153 raft_consensus.cc:2258] T 99e5feeda1eb4affb2532bda1b0055cd P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:49.159315 27153 ts_tablet_manager.cc:1905] T 99e5feeda1eb4affb2532bda1b0055cd P 4fd6381544574ddca559ce63b9243efc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 11:09:49.174045 27153 ts_tablet_manager.cc:1918] T 99e5feeda1eb4affb2532bda1b0055cd P 4fd6381544574ddca559ce63b9243efc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 11:09:49.174490 27153 log.cc:1196] T 99e5feeda1eb4affb2532bda1b0055cd P 4fd6381544574ddca559ce63b9243efc: Deleting WAL directory at /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-1-root/wals/99e5feeda1eb4affb2532bda1b0055cd
I20260807 11:09:49.175827 25637 catalog_manager.cc:4841] TS 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): tablet 99e5feeda1eb4affb2532bda1b0055cd (table test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]) successfully deleted
I20260807 11:09:50.158893 25926 consensus_queue.cc:237] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:50.167436 25786 raft_consensus.cc:1261] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:50.168684 26038 raft_consensus.cc:1261] T f9465d2caa3e43c8891f52350177df8d P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 11:09:50.169538 25869 consensus_peers.cc:487] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9465d2caa3e43c8891f52350177df8d. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:50.170466 27047 consensus_queue.cc:1035] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:50.172067 27163 consensus_queue.cc:1035] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:09:50.182224 27047 raft_consensus.cc:2932] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:50.187085 26046 raft_consensus.cc:2932] T f9465d2caa3e43c8891f52350177df8d P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:50.189294 25786 raft_consensus.cc:2932] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:50.209669 25660 catalog_manager.cc:5495] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:50.238312 26038 consensus_queue.cc:237] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:50.247992 25926 raft_consensus.cc:1261] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:50.250337 25786 raft_consensus.cc:1261] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:50.251082 26860 consensus_queue.cc:1035] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:50.252616 27155 consensus_queue.cc:1035] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:09:50.269335 27183 raft_consensus.cc:2932] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
W20260807 11:09:50.276819 25981 consensus_peers.cc:487] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d47975fbb6e3412aaf146ed0794f986c. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:50.277390 25919 raft_consensus.cc:2932] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:50.297300 25800 raft_consensus.cc:2932] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:50.303614 25919 consensus_queue.cc:237] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:50.305894 25654 catalog_manager.cc:5495] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:50.325510 26038 raft_consensus.cc:1261] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:50.327138 27162 consensus_queue.cc:1035] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:50.329317 25800 raft_consensus.cc:1261] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 11:09:50.331117 25869 consensus_peers.cc:487] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b7c44ad3ac214b2d83c41bc5aa01f349. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:50.331131 27113 consensus_queue.cc:1035] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:09:50.337937 27162 raft_consensus.cc:2932] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:50.344616 25800 raft_consensus.cc:2932] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:50.339495 26038 raft_consensus.cc:2932] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:50.355392 25919 consensus_queue.cc:237] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:50.360416 25659 catalog_manager.cc:5495] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260807 11:09:50.375536 25869 consensus_peers.cc:487] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f35a17e1683e41f4b146284f5cf09c58. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:50.377182 26038 raft_consensus.cc:1261] T f35a17e1683e41f4b146284f5cf09c58 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:50.378559 27113 consensus_queue.cc:1035] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:50.381093 25786 raft_consensus.cc:1261] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:50.382519 27113 consensus_queue.cc:1035] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:50.403421 27162 raft_consensus.cc:2932] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:50.405122 25800 raft_consensus.cc:2932] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:50.408006 26046 raft_consensus.cc:2932] T f35a17e1683e41f4b146284f5cf09c58 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:50.428153 25654 catalog_manager.cc:5495] T f35a17e1683e41f4b146284f5cf09c58 P b22d69db45194908ad3d345eba9badd9 reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:50.637874 27204 ts_tablet_manager.cc:927] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:50.640847 27204 tablet_copy_client.cc:316] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.130:36465
I20260807 11:09:50.642446 25945 tablet_copy_service.cc:140] P 4fd6381544574ddca559ce63b9243efc: Received BeginTabletCopySession request for tablet f9465d2caa3e43c8891f52350177df8d from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:35276)
I20260807 11:09:50.642968 25945 tablet_copy_service.cc:161] P 4fd6381544574ddca559ce63b9243efc: Beginning new tablet copy session on tablet f9465d2caa3e43c8891f52350177df8d from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:35276: session id = eb1fca62dcb34ed9b32188ccb19f0543-f9465d2caa3e43c8891f52350177df8d
I20260807 11:09:50.650067 25945 tablet_copy_source_session.cc:205] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:09:50.655460 27204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9465d2caa3e43c8891f52350177df8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:50.676873 27204 tablet_copy_client.cc:791] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:09:50.677613 27204 tablet_copy_client.cc:655] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:09:50.683347 27204 tablet_copy_client.cc:531] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:09:50.691941 27204 tablet_bootstrap.cc:492] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
I20260807 11:09:50.712306 27204 tablet_bootstrap.cc:492] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:09:50.713230 27204 tablet_bootstrap.cc:492] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:09:50.713973 27204 ts_tablet_manager.cc:1397] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.010s
I20260807 11:09:50.715025 27209 ts_tablet_manager.cc:927] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:50.716454 27204 raft_consensus.cc:348] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:50.717238 27209 tablet_copy_client.cc:316] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.131:46009
I20260807 11:09:50.717604 27204 raft_consensus.cc:729] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:09:50.718374 27204 consensus_queue.cc:260] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:50.721825 27204 ts_tablet_manager.cc:1428] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.008s user 0.006s sys 0.002s
I20260807 11:09:50.740345 25945 tablet_copy_service.cc:342] P 4fd6381544574ddca559ce63b9243efc: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-f9465d2caa3e43c8891f52350177df8d received from {username='slave'} at 127.0.0.1:35276
I20260807 11:09:50.740746 25945 tablet_copy_service.cc:433] P 4fd6381544574ddca559ce63b9243efc: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-f9465d2caa3e43c8891f52350177df8d on tablet f9465d2caa3e43c8891f52350177df8d with peer eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:09:50.751058 26061 tablet_copy_service.cc:140] P b22d69db45194908ad3d345eba9badd9: Received BeginTabletCopySession request for tablet d47975fbb6e3412aaf146ed0794f986c from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:43246)
I20260807 11:09:50.751636 26061 tablet_copy_service.cc:161] P b22d69db45194908ad3d345eba9badd9: Beginning new tablet copy session on tablet d47975fbb6e3412aaf146ed0794f986c from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:43246: session id = eb1fca62dcb34ed9b32188ccb19f0543-d47975fbb6e3412aaf146ed0794f986c
I20260807 11:09:50.759451 26061 tablet_copy_source_session.cc:205] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:09:50.762460 27209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d47975fbb6e3412aaf146ed0794f986c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:50.777822 27209 tablet_copy_client.cc:791] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:09:50.779997 27209 tablet_copy_client.cc:655] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:09:50.783600 27209 tablet_copy_client.cc:531] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:09:50.790438 27209 tablet_bootstrap.cc:492] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
I20260807 11:09:50.810429 27209 tablet_bootstrap.cc:492] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:09:50.811229 27209 tablet_bootstrap.cc:492] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:09:50.811949 27209 ts_tablet_manager.cc:1397] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.008s
I20260807 11:09:50.814651 27209 raft_consensus.cc:348] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:50.815369 27209 raft_consensus.cc:729] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:09:50.816036 27209 consensus_queue.cc:260] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:50.818463 27209 ts_tablet_manager.cc:1428] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260807 11:09:50.820845 26061 tablet_copy_service.cc:342] P b22d69db45194908ad3d345eba9badd9: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-d47975fbb6e3412aaf146ed0794f986c received from {username='slave'} at 127.0.0.1:43246
I20260807 11:09:50.821265 26061 tablet_copy_service.cc:433] P b22d69db45194908ad3d345eba9badd9: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-d47975fbb6e3412aaf146ed0794f986c on tablet d47975fbb6e3412aaf146ed0794f986c with peer eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:09:50.828328 27209 ts_tablet_manager.cc:927] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:50.830170 27209 tablet_copy_client.cc:316] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.130:36465
I20260807 11:09:50.832270 25945 tablet_copy_service.cc:140] P 4fd6381544574ddca559ce63b9243efc: Received BeginTabletCopySession request for tablet b7c44ad3ac214b2d83c41bc5aa01f349 from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:35276)
I20260807 11:09:50.832857 25945 tablet_copy_service.cc:161] P 4fd6381544574ddca559ce63b9243efc: Beginning new tablet copy session on tablet b7c44ad3ac214b2d83c41bc5aa01f349 from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:35276: session id = eb1fca62dcb34ed9b32188ccb19f0543-b7c44ad3ac214b2d83c41bc5aa01f349
I20260807 11:09:50.841244 25945 tablet_copy_source_session.cc:205] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:09:50.844143 27209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7c44ad3ac214b2d83c41bc5aa01f349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:50.859301 27209 tablet_copy_client.cc:791] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:09:50.859896 27209 tablet_copy_client.cc:655] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:09:50.863806 27209 tablet_copy_client.cc:531] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:09:50.870805 27209 tablet_bootstrap.cc:492] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
I20260807 11:09:50.893823 27209 tablet_bootstrap.cc:492] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:09:50.894867 27209 tablet_bootstrap.cc:492] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:09:50.895573 27209 ts_tablet_manager.cc:1397] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.009s
I20260807 11:09:50.898288 27209 raft_consensus.cc:348] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:50.899231 27209 raft_consensus.cc:729] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:09:50.899926 27209 consensus_queue.cc:260] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:50.905508 27209 ts_tablet_manager.cc:1428] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.010s user 0.003s sys 0.003s
I20260807 11:09:50.907567 25945 tablet_copy_service.cc:342] P 4fd6381544574ddca559ce63b9243efc: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-b7c44ad3ac214b2d83c41bc5aa01f349 received from {username='slave'} at 127.0.0.1:35276
I20260807 11:09:50.908082 25945 tablet_copy_service.cc:433] P 4fd6381544574ddca559ce63b9243efc: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-b7c44ad3ac214b2d83c41bc5aa01f349 on tablet b7c44ad3ac214b2d83c41bc5aa01f349 with peer eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:09:50.990419 27209 ts_tablet_manager.cc:927] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:50.992118 27209 tablet_copy_client.cc:316] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.130:36465
I20260807 11:09:50.993798 25945 tablet_copy_service.cc:140] P 4fd6381544574ddca559ce63b9243efc: Received BeginTabletCopySession request for tablet f35a17e1683e41f4b146284f5cf09c58 from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:35276)
I20260807 11:09:50.994271 25945 tablet_copy_service.cc:161] P 4fd6381544574ddca559ce63b9243efc: Beginning new tablet copy session on tablet f35a17e1683e41f4b146284f5cf09c58 from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:35276: session id = eb1fca62dcb34ed9b32188ccb19f0543-f35a17e1683e41f4b146284f5cf09c58
I20260807 11:09:51.002110 25945 tablet_copy_source_session.cc:205] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:09:51.004945 27209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f35a17e1683e41f4b146284f5cf09c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:51.019286 27209 tablet_copy_client.cc:791] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:09:51.019896 27209 tablet_copy_client.cc:655] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:09:51.024942 27209 tablet_copy_client.cc:531] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:09:51.033363 27209 tablet_bootstrap.cc:492] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
I20260807 11:09:51.049180 27209 tablet_bootstrap.cc:492] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:09:51.049875 27209 tablet_bootstrap.cc:492] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:09:51.050462 27209 ts_tablet_manager.cc:1397] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20260807 11:09:51.052685 27209 raft_consensus.cc:348] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:51.053527 27209 raft_consensus.cc:729] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:09:51.054075 27209 consensus_queue.cc:260] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:51.058894 27209 ts_tablet_manager.cc:1428] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260807 11:09:51.060526 25945 tablet_copy_service.cc:342] P 4fd6381544574ddca559ce63b9243efc: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-f35a17e1683e41f4b146284f5cf09c58 received from {username='slave'} at 127.0.0.1:35276
I20260807 11:09:51.060968 25945 tablet_copy_service.cc:433] P 4fd6381544574ddca559ce63b9243efc: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-f35a17e1683e41f4b146284f5cf09c58 on tablet f35a17e1683e41f4b146284f5cf09c58 with peer eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:09:51.177070 26987 raft_consensus.cc:1203] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:09:51.195469 26987 raft_consensus.cc:1203] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:09:51.278350 26987 raft_consensus.cc:1203] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:09:51.514701 26986 raft_consensus.cc:1203] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:09:51.611599 27163 raft_consensus.cc:1050] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:09:51.613865 27163 consensus_queue.cc:237] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:51.621302 26986 raft_consensus.cc:1261] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:51.621402 26038 raft_consensus.cc:1261] T f9465d2caa3e43c8891f52350177df8d P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:51.623028 27162 consensus_queue.cc:1035] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:51.629396 27162 consensus_queue.cc:1035] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:51.633960 25800 raft_consensus.cc:1261] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:51.635615 27221 consensus_queue.cc:1035] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:51.639703 27163 raft_consensus.cc:2932] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:51.641788 26986 raft_consensus.cc:2932] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:51.652607 26038 raft_consensus.cc:2932] T f9465d2caa3e43c8891f52350177df8d P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:51.658002 25659 catalog_manager.cc:5495] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:51.673990 25800 raft_consensus.cc:2932] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:51.710371 25926 consensus_queue.cc:237] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:51.715498 25800 raft_consensus.cc:1261] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:51.716915 27221 consensus_queue.cc:1035] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:09:51.716655 26986 raft_consensus.cc:1261] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:51.718811 27221 consensus_queue.cc:1035] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:09:51.725356 27047 raft_consensus.cc:1050] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:09:51.728147 27217 raft_consensus.cc:1050] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:09:51.730324 27217 consensus_queue.cc:237] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:51.727629 27047 consensus_queue.cc:237] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:51.738420 27221 raft_consensus.cc:2932] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:51.743574 26987 raft_consensus.cc:1261] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:51.744748 25919 raft_consensus.cc:1261] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:51.746163 27187 consensus_queue.cc:1035] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:51.750509 26860 consensus_queue.cc:1035] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:51.752948 25637 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet f9465d2caa3e43c8891f52350177df8d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 11:09:51.754478 25800 raft_consensus.cc:2932] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:51.758504 25801 raft_consensus.cc:1261] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:51.759843 25786 raft_consensus.cc:1261] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:51.761118 26038 raft_consensus.cc:1261] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:51.762388 27047 consensus_queue.cc:1035] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:51.763146 27162 consensus_queue.cc:1035] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:51.765057 26985 raft_consensus.cc:1261] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:51.775424 26860 consensus_queue.cc:1035] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:51.772485 25660 catalog_manager.cc:5495] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc reported cstate change: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:51.788472 27162 consensus_queue.cc:1035] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:09:51.793979 26032 tablet_service.cc:1514] Processing DeleteTablet for tablet f9465d2caa3e43c8891f52350177df8d with delete_type TABLET_DATA_TOMBSTONED (TS b22d69db45194908ad3d345eba9badd9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59300
I20260807 11:09:51.796990 26985 raft_consensus.cc:2932] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:51.827584 27248 tablet_replica.cc:330] T f9465d2caa3e43c8891f52350177df8d P b22d69db45194908ad3d345eba9badd9: stopping tablet replica
I20260807 11:09:51.836949 27248 raft_consensus.cc:2229] T f9465d2caa3e43c8891f52350177df8d P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:51.835911 26860 raft_consensus.cc:2932] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:51.837667 27248 raft_consensus.cc:2258] T f9465d2caa3e43c8891f52350177df8d P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:51.842873 25799 raft_consensus.cc:2932] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:51.844398 27237 raft_consensus.cc:2932] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:51.847769 26986 raft_consensus.cc:2932] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:51.853080 25919 raft_consensus.cc:2932] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:51.859575 25801 raft_consensus.cc:2932] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:51.865664 27248 ts_tablet_manager.cc:1905] T f9465d2caa3e43c8891f52350177df8d P b22d69db45194908ad3d345eba9badd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 11:09:51.867036 25660 catalog_manager.cc:5495] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:51.877492 26986 raft_consensus.cc:2932] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:51.898295 27248 ts_tablet_manager.cc:1918] T f9465d2caa3e43c8891f52350177df8d P b22d69db45194908ad3d345eba9badd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 11:09:51.897943 25660 catalog_manager.cc:5495] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:51.899351 27248 log.cc:1196] T f9465d2caa3e43c8891f52350177df8d P b22d69db45194908ad3d345eba9badd9: Deleting WAL directory at /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-2-root/wals/f9465d2caa3e43c8891f52350177df8d
I20260807 11:09:51.903712 26046 raft_consensus.cc:2932] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:51.910086 25639 catalog_manager.cc:4841] TS b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): tablet f9465d2caa3e43c8891f52350177df8d (table test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]) successfully deleted
I20260807 11:09:51.912230 26038 consensus_queue.cc:237] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:51.929252 25919 raft_consensus.cc:1261] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:51.930698 27240 consensus_queue.cc:1035] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:09:51.934242 26986 raft_consensus.cc:1261] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:51.935817 27240 consensus_queue.cc:1035] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20260807 11:09:51.941341 25919 consensus_queue.cc:237] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:51.944536 26860 raft_consensus.cc:2932] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:51.948084 26985 raft_consensus.cc:1261] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:51.949801 26038 raft_consensus.cc:1261] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:51.952131 27237 consensus_queue.cc:1035] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:09:51.950858 25919 raft_consensus.cc:2932] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:51.959879 27237 consensus_queue.cc:1035] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:09:51.961018 26985 raft_consensus.cc:2932] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:51.965343 27221 raft_consensus.cc:2932] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:51.968631 26038 raft_consensus.cc:2932] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:51.982226 25659 catalog_manager.cc:5495] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc reported cstate change: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:51.984048 25639 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d47975fbb6e3412aaf146ed0794f986c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 11:09:51.989601 25637 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet b7c44ad3ac214b2d83c41bc5aa01f349 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 11:09:52.006906 26986 raft_consensus.cc:2932] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:52.041635 25660 catalog_manager.cc:5495] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 reported cstate change: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:52.042850 25785 tablet_service.cc:1514] Processing DeleteTablet for tablet d47975fbb6e3412aaf146ed0794f986c with delete_type TABLET_DATA_TOMBSTONED (TS 606d49e053a346e486f48cb31832b815 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53306
I20260807 11:09:52.055331 27253 tablet_replica.cc:330] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815: stopping tablet replica
I20260807 11:09:52.057075 27253 raft_consensus.cc:2229] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:52.058431 27253 raft_consensus.cc:2258] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:52.063217 27253 ts_tablet_manager.cc:1905] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 11:09:52.082084 27253 ts_tablet_manager.cc:1918] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 11:09:52.082728 27253 log.cc:1196] T d47975fbb6e3412aaf146ed0794f986c P 606d49e053a346e486f48cb31832b815: Deleting WAL directory at /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-0-root/wals/d47975fbb6e3412aaf146ed0794f986c
I20260807 11:09:52.088275 25785 tablet_service.cc:1514] Processing DeleteTablet for tablet b7c44ad3ac214b2d83c41bc5aa01f349 with delete_type TABLET_DATA_TOMBSTONED (TS 606d49e053a346e486f48cb31832b815 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53306
I20260807 11:09:52.090222 27253 tablet_replica.cc:330] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815: stopping tablet replica
I20260807 11:09:52.091471 25637 catalog_manager.cc:4841] TS 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): tablet d47975fbb6e3412aaf146ed0794f986c (table test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]) successfully deleted
I20260807 11:09:52.091735 27253 raft_consensus.cc:2229] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:52.093982 27253 raft_consensus.cc:2258] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:52.097946 27253 ts_tablet_manager.cc:1905] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 11:09:52.120651 27253 ts_tablet_manager.cc:1918] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 11:09:52.121266 27253 log.cc:1196] T b7c44ad3ac214b2d83c41bc5aa01f349 P 606d49e053a346e486f48cb31832b815: Deleting WAL directory at /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-0-root/wals/b7c44ad3ac214b2d83c41bc5aa01f349
I20260807 11:09:52.123775 25637 catalog_manager.cc:4841] TS 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): tablet b7c44ad3ac214b2d83c41bc5aa01f349 (table test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]) successfully deleted
I20260807 11:09:52.137534 27221 raft_consensus.cc:1050] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:09:52.140856 27221 consensus_queue.cc:237] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:52.149403 25801 raft_consensus.cc:1261] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:52.149582 26986 raft_consensus.cc:1261] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:52.150990 27047 consensus_queue.cc:1035] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260807 11:09:52.153131 27162 consensus_queue.cc:1035] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:09:52.158460 26038 raft_consensus.cc:1261] T f35a17e1683e41f4b146284f5cf09c58 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:52.160724 27162 consensus_queue.cc:1035] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:09:52.169579 27236 raft_consensus.cc:2932] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:52.174047 26986 raft_consensus.cc:2932] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:52.193527 25801 raft_consensus.cc:2932] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:52.201354 26033 raft_consensus.cc:2932] T f35a17e1683e41f4b146284f5cf09c58 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:52.201699 25654 catalog_manager.cc:5495] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:09:52.230248 25917 consensus_queue.cc:237] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:52.237775 25801 raft_consensus.cc:1261] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:52.236338 26986 raft_consensus.cc:1261] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:52.240564 27236 consensus_queue.cc:1035] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260807 11:09:52.243666 27047 consensus_queue.cc:1035] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:09:52.253899 27047 raft_consensus.cc:2932] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:52.257221 25801 raft_consensus.cc:2932] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:52.257939 26986 raft_consensus.cc:2932] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:52.283999 25654 catalog_manager.cc:5495] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 reported cstate change: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:52.289873 25637 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet f35a17e1683e41f4b146284f5cf09c58 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 11:09:52.300678 26032 tablet_service.cc:1514] Processing DeleteTablet for tablet f35a17e1683e41f4b146284f5cf09c58 with delete_type TABLET_DATA_TOMBSTONED (TS b22d69db45194908ad3d345eba9badd9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59300
I20260807 11:09:52.302611 27248 tablet_replica.cc:330] T f35a17e1683e41f4b146284f5cf09c58 P b22d69db45194908ad3d345eba9badd9: stopping tablet replica
I20260807 11:09:52.309226 27248 raft_consensus.cc:2229] T f35a17e1683e41f4b146284f5cf09c58 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:52.310200 27248 raft_consensus.cc:2258] T f35a17e1683e41f4b146284f5cf09c58 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:52.314154 27248 ts_tablet_manager.cc:1905] T f35a17e1683e41f4b146284f5cf09c58 P b22d69db45194908ad3d345eba9badd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 11:09:52.333384 27248 ts_tablet_manager.cc:1918] T f35a17e1683e41f4b146284f5cf09c58 P b22d69db45194908ad3d345eba9badd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 11:09:52.333820 27248 log.cc:1196] T f35a17e1683e41f4b146284f5cf09c58 P b22d69db45194908ad3d345eba9badd9: Deleting WAL directory at /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-2-root/wals/f35a17e1683e41f4b146284f5cf09c58
I20260807 11:09:52.335345 25639 catalog_manager.cc:4841] TS b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): tablet f35a17e1683e41f4b146284f5cf09c58 (table test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]) successfully deleted
W20260807 11:09:53.410591 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 72036.5us
I20260807 11:09:53.543722 25801 consensus_queue.cc:237] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
W20260807 11:09:53.555122 25750 consensus_peers.cc:487] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 986fe968b7be4eab9c480ec85c0384b0. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:53.556305 26038 raft_consensus.cc:1261] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:53.558791 25926 raft_consensus.cc:1261] T 986fe968b7be4eab9c480ec85c0384b0 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:53.560055 27279 consensus_queue.cc:1035] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:09:53.562458 27157 consensus_queue.cc:1035] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:09:53.573964 27280 raft_consensus.cc:2932] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:53.574625 26038 raft_consensus.cc:2932] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:53.577710 25926 raft_consensus.cc:2932] T 986fe968b7be4eab9c480ec85c0384b0 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:53.615190 25919 consensus_queue.cc:237] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:53.619025 25659 catalog_manager.cc:5495] T 986fe968b7be4eab9c480ec85c0384b0 P 4fd6381544574ddca559ce63b9243efc reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
W20260807 11:09:53.631505 25869 consensus_peers.cc:487] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e702bb04fc98457f9d8ee704fcaeafcf. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:53.631652 26038 raft_consensus.cc:1261] T e702bb04fc98457f9d8ee704fcaeafcf P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:53.632460 25801 raft_consensus.cc:1261] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:53.633664 27286 consensus_queue.cc:1035] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:09:53.635038 27276 consensus_queue.cc:1035] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:09:53.652427 27286 raft_consensus.cc:2932] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:53.654228 25801 raft_consensus.cc:2932] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:53.661101 26038 raft_consensus.cc:2932] T e702bb04fc98457f9d8ee704fcaeafcf P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:53.680794 26038 consensus_queue.cc:237] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:53.688858 25919 raft_consensus.cc:1261] T 6eed5249afac4e16a4b17938ad6e1353 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:53.691838 27277 consensus_queue.cc:1035] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:09:53.692579 25801 raft_consensus.cc:1261] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 11:09:53.702176 25981 consensus_peers.cc:487] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6eed5249afac4e16a4b17938ad6e1353. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:53.702639 27251 consensus_queue.cc:1035] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:09:53.708221 25654 catalog_manager.cc:5495] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:53.720638 27277 raft_consensus.cc:2932] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:53.722213 25801 raft_consensus.cc:2932] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:53.724802 25919 raft_consensus.cc:2932] T 6eed5249afac4e16a4b17938ad6e1353 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:53.743534 26038 consensus_queue.cc:237] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:53.743728 25659 catalog_manager.cc:5495] T 6eed5249afac4e16a4b17938ad6e1353 P 4fd6381544574ddca559ce63b9243efc reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
W20260807 11:09:53.757018 25981 consensus_peers.cc:487] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0fc5b1dfff1a4556abdd562afbebd768. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:53.761435 25801 raft_consensus.cc:1261] T 0fc5b1dfff1a4556abdd562afbebd768 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:53.763013 27251 consensus_queue.cc:1035] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:09:53.765434 25917 raft_consensus.cc:1261] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:53.768740 27251 consensus_queue.cc:1035] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:53.788617 27251 raft_consensus.cc:2932] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:53.791280 25917 raft_consensus.cc:2932] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:53.792153 25801 raft_consensus.cc:2932] T 0fc5b1dfff1a4556abdd562afbebd768 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:53.808712 25654 catalog_manager.cc:5495] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:54.066390 27312 ts_tablet_manager.cc:927] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195)
I20260807 11:09:54.068550 27312 tablet_copy_client.cc:316] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.129:40195
I20260807 11:09:54.070431 25825 tablet_copy_service.cc:140] P 606d49e053a346e486f48cb31832b815: Received BeginTabletCopySession request for tablet 986fe968b7be4eab9c480ec85c0384b0 from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:59118)
I20260807 11:09:54.071004 25825 tablet_copy_service.cc:161] P 606d49e053a346e486f48cb31832b815: Beginning new tablet copy session on tablet 986fe968b7be4eab9c480ec85c0384b0 from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:59118: session id = eb1fca62dcb34ed9b32188ccb19f0543-986fe968b7be4eab9c480ec85c0384b0
I20260807 11:09:54.080415 25825 tablet_copy_source_session.cc:205] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:09:54.087967 27312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 986fe968b7be4eab9c480ec85c0384b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:54.103351 27312 tablet_copy_client.cc:791] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:09:54.104151 27312 tablet_copy_client.cc:655] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:09:54.108901 27312 tablet_copy_client.cc:531] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:09:54.119382 27312 tablet_bootstrap.cc:492] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
I20260807 11:09:54.147220 27312 tablet_bootstrap.cc:492] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:09:54.149536 27312 tablet_bootstrap.cc:492] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:09:54.151544 27312 ts_tablet_manager.cc:1397] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 0.032s user 0.018s sys 0.005s
I20260807 11:09:54.154861 27312 raft_consensus.cc:348] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:54.155786 27312 raft_consensus.cc:729] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:09:54.156412 27312 consensus_queue.cc:260] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:54.161592 27312 ts_tablet_manager.cc:1428] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260807 11:09:54.164570 25825 tablet_copy_service.cc:342] P 606d49e053a346e486f48cb31832b815: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-986fe968b7be4eab9c480ec85c0384b0 received from {username='slave'} at 127.0.0.1:59118
I20260807 11:09:54.165069 25825 tablet_copy_service.cc:433] P 606d49e053a346e486f48cb31832b815: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-986fe968b7be4eab9c480ec85c0384b0 on tablet 986fe968b7be4eab9c480ec85c0384b0 with peer eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:09:54.206807 27312 ts_tablet_manager.cc:927] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:09:54.208252 27312 tablet_copy_client.cc:316] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.130:36465
I20260807 11:09:54.210153 25945 tablet_copy_service.cc:140] P 4fd6381544574ddca559ce63b9243efc: Received BeginTabletCopySession request for tablet e702bb04fc98457f9d8ee704fcaeafcf from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:35276)
I20260807 11:09:54.210819 25945 tablet_copy_service.cc:161] P 4fd6381544574ddca559ce63b9243efc: Beginning new tablet copy session on tablet e702bb04fc98457f9d8ee704fcaeafcf from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:35276: session id = eb1fca62dcb34ed9b32188ccb19f0543-e702bb04fc98457f9d8ee704fcaeafcf
I20260807 11:09:54.220280 25945 tablet_copy_source_session.cc:205] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:09:54.223593 27312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e702bb04fc98457f9d8ee704fcaeafcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:54.239399 27312 tablet_copy_client.cc:791] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:09:54.239578 27320 ts_tablet_manager.cc:927] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:54.239966 27312 tablet_copy_client.cc:655] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:09:54.257342 27320 tablet_copy_client.cc:316] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.131:46009
I20260807 11:09:54.259363 26061 tablet_copy_service.cc:140] P b22d69db45194908ad3d345eba9badd9: Received BeginTabletCopySession request for tablet 6eed5249afac4e16a4b17938ad6e1353 from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:43246)
I20260807 11:09:54.259935 26061 tablet_copy_service.cc:161] P b22d69db45194908ad3d345eba9badd9: Beginning new tablet copy session on tablet 6eed5249afac4e16a4b17938ad6e1353 from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:43246: session id = eb1fca62dcb34ed9b32188ccb19f0543-6eed5249afac4e16a4b17938ad6e1353
I20260807 11:09:54.264460 27312 tablet_copy_client.cc:531] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:09:54.270560 26061 tablet_copy_source_session.cc:205] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:09:54.274282 27312 tablet_bootstrap.cc:492] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
I20260807 11:09:54.286898 27320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eed5249afac4e16a4b17938ad6e1353. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:54.306315 27322 ts_tablet_manager.cc:927] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:54.372736 27320 tablet_copy_client.cc:791] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:09:54.373558 27320 tablet_copy_client.cc:655] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:09:54.377530 27320 tablet_copy_client.cc:531] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:09:54.393798 27320 tablet_bootstrap.cc:492] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
I20260807 11:09:54.408449 27312 tablet_bootstrap.cc:492] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:09:54.410948 27312 tablet_bootstrap.cc:492] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:09:54.419242 27312 ts_tablet_manager.cc:1397] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 0.145s user 0.047s sys 0.046s
I20260807 11:09:54.425551 27320 tablet_bootstrap.cc:492] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:09:54.428387 27320 tablet_bootstrap.cc:492] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:09:54.429142 27320 ts_tablet_manager.cc:1397] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 0.036s user 0.022s sys 0.006s
I20260807 11:09:54.432629 27312 raft_consensus.cc:348] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:54.432830 27320 raft_consensus.cc:348] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:54.433702 27312 raft_consensus.cc:729] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:09:54.433841 27320 raft_consensus.cc:729] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:09:54.434441 27312 consensus_queue.cc:260] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:54.435706 27320 consensus_queue.cc:260] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:54.437005 27312 ts_tablet_manager.cc:1428] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260807 11:09:54.441623 25945 tablet_copy_service.cc:342] P 4fd6381544574ddca559ce63b9243efc: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-e702bb04fc98457f9d8ee704fcaeafcf received from {username='slave'} at 127.0.0.1:35276
I20260807 11:09:54.442049 25945 tablet_copy_service.cc:433] P 4fd6381544574ddca559ce63b9243efc: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-e702bb04fc98457f9d8ee704fcaeafcf on tablet e702bb04fc98457f9d8ee704fcaeafcf with peer eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:09:54.442243 27322 tablet_copy_client.cc:316] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.131:46009
I20260807 11:09:54.445731 26061 tablet_copy_service.cc:140] P b22d69db45194908ad3d345eba9badd9: Received BeginTabletCopySession request for tablet 0fc5b1dfff1a4556abdd562afbebd768 from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:43246)
I20260807 11:09:54.446405 26061 tablet_copy_service.cc:161] P b22d69db45194908ad3d345eba9badd9: Beginning new tablet copy session on tablet 0fc5b1dfff1a4556abdd562afbebd768 from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:43246: session id = eb1fca62dcb34ed9b32188ccb19f0543-0fc5b1dfff1a4556abdd562afbebd768
I20260807 11:09:54.451215 27320 ts_tablet_manager.cc:1428] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260807 11:09:54.452940 26060 tablet_copy_service.cc:342] P b22d69db45194908ad3d345eba9badd9: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-6eed5249afac4e16a4b17938ad6e1353 received from {username='slave'} at 127.0.0.1:43246
I20260807 11:09:54.453336 26060 tablet_copy_service.cc:433] P b22d69db45194908ad3d345eba9badd9: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-6eed5249afac4e16a4b17938ad6e1353 on tablet 6eed5249afac4e16a4b17938ad6e1353 with peer eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:09:54.456043 26061 tablet_copy_source_session.cc:205] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:09:54.460098 27322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc5b1dfff1a4556abdd562afbebd768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:54.478070 27322 tablet_copy_client.cc:791] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:09:54.478811 27322 tablet_copy_client.cc:655] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:09:54.486172 27322 tablet_copy_client.cc:531] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:09:54.494379 27322 tablet_bootstrap.cc:492] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
I20260807 11:09:54.514110 27322 tablet_bootstrap.cc:492] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:09:54.515029 27322 tablet_bootstrap.cc:492] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:09:54.516003 27322 ts_tablet_manager.cc:1397] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.013s
I20260807 11:09:54.518841 27322 raft_consensus.cc:348] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:54.519680 27322 raft_consensus.cc:729] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:09:54.520402 27322 consensus_queue.cc:260] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:54.523252 27322 ts_tablet_manager.cc:1428] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260807 11:09:54.529292 26061 tablet_copy_service.cc:342] P b22d69db45194908ad3d345eba9badd9: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-0fc5b1dfff1a4556abdd562afbebd768 received from {username='slave'} at 127.0.0.1:43246
I20260807 11:09:54.529762 26061 tablet_copy_service.cc:433] P b22d69db45194908ad3d345eba9badd9: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-0fc5b1dfff1a4556abdd562afbebd768 on tablet 0fc5b1dfff1a4556abdd562afbebd768 with peer eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:09:54.591975 26986 raft_consensus.cc:1203] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:09:54.721354 26986 raft_consensus.cc:1203] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:09:54.750319 26986 raft_consensus.cc:1203] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:09:54.788998 26986 raft_consensus.cc:1203] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:09:55.117076 27143 raft_consensus.cc:1050] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:09:55.119091 27143 consensus_queue.cc:237] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:55.124105 26986 raft_consensus.cc:1261] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:55.125186 25917 raft_consensus.cc:1261] T 986fe968b7be4eab9c480ec85c0384b0 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:55.125520 27278 consensus_queue.cc:1035] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:55.127621 26038 raft_consensus.cc:1261] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:55.127926 27278 consensus_queue.cc:1035] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:55.132596 27143 consensus_queue.cc:1035] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:55.151124 27143 raft_consensus.cc:2932] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.153447 25926 raft_consensus.cc:2932] T 986fe968b7be4eab9c480ec85c0384b0 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.154633 26986 raft_consensus.cc:2932] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.173080 25659 catalog_manager.cc:5495] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:09:55.174418 27327 raft_consensus.cc:1050] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:09:55.176486 26033 raft_consensus.cc:2932] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.186511 27327 consensus_queue.cc:237] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:55.196645 26986 raft_consensus.cc:1261] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:55.198177 27221 consensus_queue.cc:1035] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:09:55.200150 26046 raft_consensus.cc:1261] T e702bb04fc98457f9d8ee704fcaeafcf P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:55.201942 27221 consensus_queue.cc:1035] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:55.215849 25800 raft_consensus.cc:1261] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:55.217640 27286 consensus_queue.cc:1035] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:55.220908 27327 raft_consensus.cc:2932] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.223726 26033 raft_consensus.cc:2932] T e702bb04fc98457f9d8ee704fcaeafcf P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.243458 25786 raft_consensus.cc:2932] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.249779 26985 raft_consensus.cc:2932] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.268607 25800 consensus_queue.cc:237] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:55.273239 26860 raft_consensus.cc:1050] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:09:55.276991 26860 consensus_queue.cc:237] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:55.279707 25659 catalog_manager.cc:5495] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.283267 27265 raft_consensus.cc:1050] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:09:55.290154 26038 raft_consensus.cc:1261] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:55.294485 27340 consensus_queue.cc:1035] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:09:55.298930 27265 consensus_queue.cc:237] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:55.305161 26986 raft_consensus.cc:1261] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:55.306630 27157 consensus_queue.cc:1035] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:09:55.321297 25926 consensus_queue.cc:237] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:55.325486 27143 raft_consensus.cc:2932] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.329335 26986 raft_consensus.cc:1261] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:55.329617 25917 raft_consensus.cc:1261] T 6eed5249afac4e16a4b17938ad6e1353 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:55.332003 25786 raft_consensus.cc:1261] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:55.332361 27355 consensus_queue.cc:1035] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:09:55.334939 27355 consensus_queue.cc:1035] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:55.341320 27251 consensus_queue.cc:1035] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:09:55.344480 26986 raft_consensus.cc:2932] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.348332 26046 raft_consensus.cc:2932] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.352807 25801 raft_consensus.cc:1261] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:55.354628 26987 raft_consensus.cc:1261] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:55.358278 26860 consensus_queue.cc:1035] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:55.361521 27286 consensus_queue.cc:1035] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260807 11:09:55.374665 26987 raft_consensus.cc:1261] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:55.361150 25799 raft_consensus.cc:1261] T 0fc5b1dfff1a4556abdd562afbebd768 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:55.378854 25926 raft_consensus.cc:1261] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:55.389829 27265 consensus_queue.cc:1035] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:55.398530 27360 consensus_queue.cc:1035] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:09:55.428544 25637 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 986fe968b7be4eab9c480ec85c0384b0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 11:09:55.434470 27221 consensus_queue.cc:1035] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:09:55.442668 25659 catalog_manager.cc:5495] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.449152 27286 raft_consensus.cc:2932] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
W20260807 11:09:55.466790 27361 log.cc:924] Time spent T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.098s user 0.001s sys 0.000s
I20260807 11:09:55.471335 25910 tablet_service.cc:1514] Processing DeleteTablet for tablet 986fe968b7be4eab9c480ec85c0384b0 with delete_type TABLET_DATA_TOMBSTONED (TS 4fd6381544574ddca559ce63b9243efc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39730
I20260807 11:09:55.493868 25800 raft_consensus.cc:2932] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.500113 25919 raft_consensus.cc:2932] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.492112 27357 raft_consensus.cc:2932] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.504382 27355 raft_consensus.cc:2932] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.539881 26986 raft_consensus.cc:2932] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.540316 25659 catalog_manager.cc:5495] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.545702 25926 raft_consensus.cc:2932] T 6eed5249afac4e16a4b17938ad6e1353 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.553396 27373 tablet_replica.cc:330] T 986fe968b7be4eab9c480ec85c0384b0 P 4fd6381544574ddca559ce63b9243efc: stopping tablet replica
I20260807 11:09:55.554234 27373 raft_consensus.cc:2229] T 986fe968b7be4eab9c480ec85c0384b0 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:55.554840 27373 raft_consensus.cc:2258] T 986fe968b7be4eab9c480ec85c0384b0 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:55.558032 27373 ts_tablet_manager.cc:1905] T 986fe968b7be4eab9c480ec85c0384b0 P 4fd6381544574ddca559ce63b9243efc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 11:09:55.562319 25637 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e702bb04fc98457f9d8ee704fcaeafcf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 11:09:55.564755 25801 raft_consensus.cc:2932] T 0fc5b1dfff1a4556abdd562afbebd768 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.581498 26036 consensus_queue.cc:237] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:55.586077 25654 catalog_manager.cc:5495] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.592427 25917 raft_consensus.cc:1261] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:55.601341 27251 consensus_queue.cc:1035] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:09:55.604416 26987 raft_consensus.cc:1261] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:55.608435 27355 consensus_queue.cc:1035] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:09:55.608229 25654 catalog_manager.cc:5495] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc reported cstate change: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 11:09:55.584585 26985 raft_consensus.cc:2932] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.622896 27251 raft_consensus.cc:2932] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.624739 26986 raft_consensus.cc:2932] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.627301 25917 raft_consensus.cc:2932] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.634555 25639 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 0fc5b1dfff1a4556abdd562afbebd768 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260807 11:09:55.637991 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 200495us
I20260807 11:09:55.644598 27373 ts_tablet_manager.cc:1918] T 986fe968b7be4eab9c480ec85c0384b0 P 4fd6381544574ddca559ce63b9243efc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 11:09:55.645248 27373 log.cc:1196] T 986fe968b7be4eab9c480ec85c0384b0 P 4fd6381544574ddca559ce63b9243efc: Deleting WAL directory at /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-1-root/wals/986fe968b7be4eab9c480ec85c0384b0
I20260807 11:09:55.662686 26032 tablet_service.cc:1514] Processing DeleteTablet for tablet e702bb04fc98457f9d8ee704fcaeafcf with delete_type TABLET_DATA_TOMBSTONED (TS b22d69db45194908ad3d345eba9badd9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59300
I20260807 11:09:55.657315 25637 catalog_manager.cc:4841] TS 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): tablet 986fe968b7be4eab9c480ec85c0384b0 (table test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]) successfully deleted
I20260807 11:09:55.664559 25801 raft_consensus.cc:2932] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.669011 26986 raft_consensus.cc:2932] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.699667 25660 catalog_manager.cc:5495] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.709762 27376 tablet_replica.cc:330] T e702bb04fc98457f9d8ee704fcaeafcf P b22d69db45194908ad3d345eba9badd9: stopping tablet replica
I20260807 11:09:55.710930 27376 raft_consensus.cc:2229] T e702bb04fc98457f9d8ee704fcaeafcf P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:55.713203 27376 raft_consensus.cc:2258] T e702bb04fc98457f9d8ee704fcaeafcf P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:55.715281 26038 consensus_queue.cc:237] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:55.721565 27376 ts_tablet_manager.cc:1905] T e702bb04fc98457f9d8ee704fcaeafcf P b22d69db45194908ad3d345eba9badd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 11:09:55.725806 26984 raft_consensus.cc:1261] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:55.727707 25785 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fc5b1dfff1a4556abdd562afbebd768 with delete_type TABLET_DATA_TOMBSTONED (TS 606d49e053a346e486f48cb31832b815 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53306
I20260807 11:09:55.727686 27355 consensus_queue.cc:1035] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:09:55.737609 25786 raft_consensus.cc:1261] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:55.739120 27355 consensus_queue.cc:1035] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:09:55.741338 27376 ts_tablet_manager.cc:1918] T e702bb04fc98457f9d8ee704fcaeafcf P b22d69db45194908ad3d345eba9badd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 11:09:55.741950 27376 log.cc:1196] T e702bb04fc98457f9d8ee704fcaeafcf P b22d69db45194908ad3d345eba9badd9: Deleting WAL directory at /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-2-root/wals/e702bb04fc98457f9d8ee704fcaeafcf
I20260807 11:09:55.743774 25639 catalog_manager.cc:4841] TS b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): tablet e702bb04fc98457f9d8ee704fcaeafcf (table test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]) successfully deleted
I20260807 11:09:55.752022 27251 raft_consensus.cc:2932] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.756223 25786 raft_consensus.cc:2932] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.759085 27377 tablet_replica.cc:330] T 0fc5b1dfff1a4556abdd562afbebd768 P 606d49e053a346e486f48cb31832b815: stopping tablet replica
I20260807 11:09:55.760346 27377 raft_consensus.cc:2229] T 0fc5b1dfff1a4556abdd562afbebd768 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:55.761472 25639 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 6eed5249afac4e16a4b17938ad6e1353 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 11:09:55.761771 27377 raft_consensus.cc:2258] T 0fc5b1dfff1a4556abdd562afbebd768 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:55.766185 26984 raft_consensus.cc:2932] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:55.770901 25659 catalog_manager.cc:5495] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 reported cstate change: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:09:55.777974 27377 ts_tablet_manager.cc:1905] T 0fc5b1dfff1a4556abdd562afbebd768 P 606d49e053a346e486f48cb31832b815: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 11:09:55.789752 25910 tablet_service.cc:1514] Processing DeleteTablet for tablet 6eed5249afac4e16a4b17938ad6e1353 with delete_type TABLET_DATA_TOMBSTONED (TS 4fd6381544574ddca559ce63b9243efc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39730
I20260807 11:09:55.792400 27373 tablet_replica.cc:330] T 6eed5249afac4e16a4b17938ad6e1353 P 4fd6381544574ddca559ce63b9243efc: stopping tablet replica
I20260807 11:09:55.793921 27373 raft_consensus.cc:2229] T 6eed5249afac4e16a4b17938ad6e1353 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:55.804169 27373 raft_consensus.cc:2258] T 6eed5249afac4e16a4b17938ad6e1353 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:55.808908 27373 ts_tablet_manager.cc:1905] T 6eed5249afac4e16a4b17938ad6e1353 P 4fd6381544574ddca559ce63b9243efc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 11:09:55.815619 27377 ts_tablet_manager.cc:1918] T 0fc5b1dfff1a4556abdd562afbebd768 P 606d49e053a346e486f48cb31832b815: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 11:09:55.816201 27377 log.cc:1196] T 0fc5b1dfff1a4556abdd562afbebd768 P 606d49e053a346e486f48cb31832b815: Deleting WAL directory at /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-0-root/wals/0fc5b1dfff1a4556abdd562afbebd768
I20260807 11:09:55.818064 25637 catalog_manager.cc:4841] TS 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): tablet 0fc5b1dfff1a4556abdd562afbebd768 (table test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]) successfully deleted
I20260807 11:09:55.827922 27373 ts_tablet_manager.cc:1918] T 6eed5249afac4e16a4b17938ad6e1353 P 4fd6381544574ddca559ce63b9243efc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 11:09:55.828486 27373 log.cc:1196] T 6eed5249afac4e16a4b17938ad6e1353 P 4fd6381544574ddca559ce63b9243efc: Deleting WAL directory at /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-1-root/wals/6eed5249afac4e16a4b17938ad6e1353
I20260807 11:09:55.830920 25637 catalog_manager.cc:4841] TS 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): tablet 6eed5249afac4e16a4b17938ad6e1353 (table test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]) successfully deleted
I20260807 11:09:56.830181 26036 consensus_queue.cc:237] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:56.837400 25919 raft_consensus.cc:1261] T b1765495856a47f09ddbb303f76c635f P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 11:09:56.837623 25981 consensus_peers.cc:487] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1765495856a47f09ddbb303f76c635f. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:56.838315 25801 raft_consensus.cc:1261] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:56.838969 27357 consensus_queue.cc:1035] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:09:56.840145 27251 consensus_queue.cc:1035] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:56.848086 27380 raft_consensus.cc:2932] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:56.850600 25801 raft_consensus.cc:2932] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:56.854669 25919 raft_consensus.cc:2932] T b1765495856a47f09ddbb303f76c635f P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:56.861812 25801 consensus_queue.cc:237] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:56.867362 25654 catalog_manager.cc:5495] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
W20260807 11:09:56.873462 25750 consensus_peers.cc:487] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 998ef34204b64d2481698cc29e055cc2. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:56.878983 25919 raft_consensus.cc:1261] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:56.875394 26036 raft_consensus.cc:1261] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:56.881078 27278 consensus_queue.cc:1035] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:56.882181 27143 consensus_queue.cc:1035] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:09:56.897475 27278 raft_consensus.cc:2932] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:56.901474 26036 raft_consensus.cc:2932] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:56.902711 25919 raft_consensus.cc:2932] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:56.924566 26038 consensus_queue.cc:237] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:56.925750 25659 catalog_manager.cc:5495] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9 reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
W20260807 11:09:56.933790 25981 consensus_peers.cc:487] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f60eb5499d64c89b971928b13bcd1be. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:56.934312 25919 raft_consensus.cc:1261] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:56.934501 25801 raft_consensus.cc:1261] T 2f60eb5499d64c89b971928b13bcd1be P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:56.938369 27357 consensus_queue.cc:1035] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:09:56.939685 27380 consensus_queue.cc:1035] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:09:57.033352 27380 raft_consensus.cc:2932] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:57.039490 25801 raft_consensus.cc:2932] T 2f60eb5499d64c89b971928b13bcd1be P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:57.043109 25919 raft_consensus.cc:2932] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:57.084946 25654 catalog_manager.cc:5495] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:57.095906 26038 consensus_queue.cc:237] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
W20260807 11:09:57.104185 25981 consensus_peers.cc:487] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: adab2949574e4c6badfcf68e51b9c072. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:09:57.106128 25919 raft_consensus.cc:1261] T adab2949574e4c6badfcf68e51b9c072 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:57.106181 25801 raft_consensus.cc:1261] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:09:57.107410 27357 consensus_queue.cc:1035] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:57.108459 27251 consensus_queue.cc:1035] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:09:57.132196 27357 raft_consensus.cc:2932] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:57.133903 25800 raft_consensus.cc:2932] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:57.146030 25917 raft_consensus.cc:2932] T adab2949574e4c6badfcf68e51b9c072 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:57.150650 25659 catalog_manager.cc:5495] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:09:57.294978 27416 ts_tablet_manager.cc:927] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:57.296996 27416 tablet_copy_client.cc:316] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.131:46009
I20260807 11:09:57.298784 26061 tablet_copy_service.cc:140] P b22d69db45194908ad3d345eba9badd9: Received BeginTabletCopySession request for tablet b1765495856a47f09ddbb303f76c635f from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:43246)
I20260807 11:09:57.299221 26061 tablet_copy_service.cc:161] P b22d69db45194908ad3d345eba9badd9: Beginning new tablet copy session on tablet b1765495856a47f09ddbb303f76c635f from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:43246: session id = eb1fca62dcb34ed9b32188ccb19f0543-b1765495856a47f09ddbb303f76c635f
I20260807 11:09:57.306373 26061 tablet_copy_source_session.cc:205] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:09:57.309500 27416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1765495856a47f09ddbb303f76c635f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:57.323623 27416 tablet_copy_client.cc:791] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:09:57.324213 27416 tablet_copy_client.cc:655] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:09:57.327823 27416 tablet_copy_client.cc:531] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:09:57.336035 27416 tablet_bootstrap.cc:492] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
I20260807 11:09:57.360528 27423 ts_tablet_manager.cc:927] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195)
I20260807 11:09:57.362103 27416 tablet_bootstrap.cc:492] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:09:57.363098 27416 tablet_bootstrap.cc:492] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:09:57.363852 27416 ts_tablet_manager.cc:1397] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 0.028s user 0.028s sys 0.001s
I20260807 11:09:57.366866 27416 raft_consensus.cc:348] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:57.367832 27416 raft_consensus.cc:729] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:09:57.368624 27416 consensus_queue.cc:260] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:57.369752 27423 tablet_copy_client.cc:316] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.129:40195
I20260807 11:09:57.371582 25825 tablet_copy_service.cc:140] P 606d49e053a346e486f48cb31832b815: Received BeginTabletCopySession request for tablet 998ef34204b64d2481698cc29e055cc2 from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:59118)
I20260807 11:09:57.372017 25825 tablet_copy_service.cc:161] P 606d49e053a346e486f48cb31832b815: Beginning new tablet copy session on tablet 998ef34204b64d2481698cc29e055cc2 from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:59118: session id = eb1fca62dcb34ed9b32188ccb19f0543-998ef34204b64d2481698cc29e055cc2
I20260807 11:09:57.372371 27416 ts_tablet_manager.cc:1428] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20260807 11:09:57.375983 26061 tablet_copy_service.cc:342] P b22d69db45194908ad3d345eba9badd9: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-b1765495856a47f09ddbb303f76c635f received from {username='slave'} at 127.0.0.1:43246
I20260807 11:09:57.376374 26061 tablet_copy_service.cc:433] P b22d69db45194908ad3d345eba9badd9: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-b1765495856a47f09ddbb303f76c635f on tablet b1765495856a47f09ddbb303f76c635f with peer eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:09:57.379829 25825 tablet_copy_source_session.cc:205] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:09:57.383157 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 998ef34204b64d2481698cc29e055cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:57.402221 27423 tablet_copy_client.cc:791] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:09:57.402841 27423 tablet_copy_client.cc:655] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:09:57.407630 27423 tablet_copy_client.cc:531] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:09:57.416697 27423 tablet_bootstrap.cc:492] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
I20260807 11:09:57.440490 27423 tablet_bootstrap.cc:492] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:09:57.449061 27423 tablet_bootstrap.cc:492] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:09:57.450788 27423 ts_tablet_manager.cc:1397] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 0.034s user 0.017s sys 0.011s
I20260807 11:09:57.458901 27423 raft_consensus.cc:348] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:57.459887 27423 raft_consensus.cc:729] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:09:57.460640 27423 consensus_queue.cc:260] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:57.462968 27423 ts_tablet_manager.cc:1428] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20260807 11:09:57.464679 25825 tablet_copy_service.cc:342] P 606d49e053a346e486f48cb31832b815: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-998ef34204b64d2481698cc29e055cc2 received from {username='slave'} at 127.0.0.1:59118
I20260807 11:09:57.465091 25825 tablet_copy_service.cc:433] P 606d49e053a346e486f48cb31832b815: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-998ef34204b64d2481698cc29e055cc2 on tablet 998ef34204b64d2481698cc29e055cc2 with peer eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:09:57.511085 27423 ts_tablet_manager.cc:927] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:57.513187 27423 tablet_copy_client.cc:316] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.131:46009
I20260807 11:09:57.514757 26061 tablet_copy_service.cc:140] P b22d69db45194908ad3d345eba9badd9: Received BeginTabletCopySession request for tablet adab2949574e4c6badfcf68e51b9c072 from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:43246)
I20260807 11:09:57.515203 26061 tablet_copy_service.cc:161] P b22d69db45194908ad3d345eba9badd9: Beginning new tablet copy session on tablet adab2949574e4c6badfcf68e51b9c072 from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:43246: session id = eb1fca62dcb34ed9b32188ccb19f0543-adab2949574e4c6badfcf68e51b9c072
I20260807 11:09:57.520149 27416 ts_tablet_manager.cc:927] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:09:57.528694 26061 tablet_copy_source_session.cc:205] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:09:57.531879 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adab2949574e4c6badfcf68e51b9c072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:57.535655 27416 tablet_copy_client.cc:316] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.131:46009
I20260807 11:09:57.537263 26061 tablet_copy_service.cc:140] P b22d69db45194908ad3d345eba9badd9: Received BeginTabletCopySession request for tablet 2f60eb5499d64c89b971928b13bcd1be from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:43246)
I20260807 11:09:57.537750 26061 tablet_copy_service.cc:161] P b22d69db45194908ad3d345eba9badd9: Beginning new tablet copy session on tablet 2f60eb5499d64c89b971928b13bcd1be from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:43246: session id = eb1fca62dcb34ed9b32188ccb19f0543-2f60eb5499d64c89b971928b13bcd1be
I20260807 11:09:57.555289 26061 tablet_copy_source_session.cc:205] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:09:57.558284 27416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f60eb5499d64c89b971928b13bcd1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:09:57.571846 27423 tablet_copy_client.cc:791] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:09:57.572454 27423 tablet_copy_client.cc:655] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:09:57.576889 27416 tablet_copy_client.cc:791] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:09:57.577719 27423 tablet_copy_client.cc:531] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:09:57.578558 27416 tablet_copy_client.cc:655] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:09:57.584228 27416 tablet_copy_client.cc:531] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:09:57.588207 27423 tablet_bootstrap.cc:492] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
I20260807 11:09:57.595548 27416 tablet_bootstrap.cc:492] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
I20260807 11:09:57.607458 27423 tablet_bootstrap.cc:492] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:09:57.608150 27423 tablet_bootstrap.cc:492] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:09:57.608695 27423 ts_tablet_manager.cc:1397] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.004s
I20260807 11:09:57.611390 27423 raft_consensus.cc:348] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:57.612287 27423 raft_consensus.cc:729] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:09:57.612996 27423 consensus_queue.cc:260] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:57.623389 27423 ts_tablet_manager.cc:1428] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260807 11:09:57.626726 27416 tablet_bootstrap.cc:492] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:09:57.627759 27416 tablet_bootstrap.cc:492] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:09:57.628494 27416 ts_tablet_manager.cc:1397] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 0.033s user 0.021s sys 0.007s
I20260807 11:09:57.631173 27416 raft_consensus.cc:348] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:57.632581 27416 raft_consensus.cc:729] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:09:57.632889 26061 tablet_copy_service.cc:342] P b22d69db45194908ad3d345eba9badd9: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-adab2949574e4c6badfcf68e51b9c072 received from {username='slave'} at 127.0.0.1:43246
I20260807 11:09:57.633342 26061 tablet_copy_service.cc:433] P b22d69db45194908ad3d345eba9badd9: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-adab2949574e4c6badfcf68e51b9c072 on tablet adab2949574e4c6badfcf68e51b9c072 with peer eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:09:57.633414 27416 consensus_queue.cc:260] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:09:57.637758 27416 ts_tablet_manager.cc:1428] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260807 11:09:57.639230 26061 tablet_copy_service.cc:342] P b22d69db45194908ad3d345eba9badd9: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-2f60eb5499d64c89b971928b13bcd1be received from {username='slave'} at 127.0.0.1:43246
I20260807 11:09:57.639596 26061 tablet_copy_service.cc:433] P b22d69db45194908ad3d345eba9badd9: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-2f60eb5499d64c89b971928b13bcd1be on tablet 2f60eb5499d64c89b971928b13bcd1be with peer eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:09:57.874230 26984 raft_consensus.cc:1203] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:09:57.904160 26984 raft_consensus.cc:1203] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:09:57.940869 26984 raft_consensus.cc:1203] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:09:58.008080 26984 raft_consensus.cc:1203] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:09:58.294148 27434 raft_consensus.cc:1050] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:09:58.296293 27434 consensus_queue.cc:237] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:58.306555 25926 raft_consensus.cc:1261] T adab2949574e4c6badfcf68e51b9c072 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:58.307317 26984 raft_consensus.cc:1261] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:58.307925 27357 consensus_queue.cc:1035] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:58.310622 25801 raft_consensus.cc:1261] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:58.312875 27406 consensus_queue.cc:1035] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:58.314292 27357 consensus_queue.cc:1035] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:58.329830 27437 raft_consensus.cc:2932] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:58.331415 26984 raft_consensus.cc:2932] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:58.343662 25917 raft_consensus.cc:2932] T adab2949574e4c6badfcf68e51b9c072 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:58.345914 25654 catalog_manager.cc:5495] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:58.350081 25800 raft_consensus.cc:2932] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:58.369098 27434 raft_consensus.cc:1050] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:09:58.374722 27434 consensus_queue.cc:237] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:58.389227 25917 raft_consensus.cc:1261] T b1765495856a47f09ddbb303f76c635f P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:58.389468 25800 raft_consensus.cc:1261] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:58.391093 27357 consensus_queue.cc:1035] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:58.392971 27357 consensus_queue.cc:1035] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:58.395254 26984 raft_consensus.cc:1261] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:58.403981 26033 consensus_queue.cc:237] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:58.404960 27357 consensus_queue.cc:1035] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:09:58.413919 26986 raft_consensus.cc:1261] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:58.414811 25800 raft_consensus.cc:1261] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:58.415117 27357 consensus_queue.cc:1035] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:09:58.416321 27437 consensus_queue.cc:1035] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:09:58.421860 27357 raft_consensus.cc:2932] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:58.427754 26986 raft_consensus.cc:2932] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:58.435065 25800 raft_consensus.cc:2932] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:58.527241 25660 catalog_manager.cc:5495] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
W20260807 11:09:58.531695 27454 log.cc:924] Time spent T b1765495856a47f09ddbb303f76c635f P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.105s user 0.000s sys 0.003s
I20260807 11:09:58.558413 25639 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet adab2949574e4c6badfcf68e51b9c072 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 11:09:58.569347 25910 tablet_service.cc:1514] Processing DeleteTablet for tablet adab2949574e4c6badfcf68e51b9c072 with delete_type TABLET_DATA_TOMBSTONED (TS 4fd6381544574ddca559ce63b9243efc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39730
I20260807 11:09:58.626430 27462 tablet_replica.cc:330] T adab2949574e4c6badfcf68e51b9c072 P 4fd6381544574ddca559ce63b9243efc: stopping tablet replica
I20260807 11:09:58.627597 27462 raft_consensus.cc:2229] T adab2949574e4c6badfcf68e51b9c072 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:58.628400 27462 raft_consensus.cc:2258] T adab2949574e4c6badfcf68e51b9c072 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:58.630600 27401 raft_consensus.cc:1050] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
W20260807 11:09:58.633597 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 46706.5us
I20260807 11:09:58.654413 27401 consensus_queue.cc:237] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:58.654749 27406 raft_consensus.cc:2932] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:58.663326 26984 raft_consensus.cc:2932] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
W20260807 11:09:58.679533 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 66549.7us
I20260807 11:09:58.682992 25926 raft_consensus.cc:2932] T b1765495856a47f09ddbb303f76c635f P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:58.689742 27462 ts_tablet_manager.cc:1905] T adab2949574e4c6badfcf68e51b9c072 P 4fd6381544574ddca559ce63b9243efc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260807 11:09:58.713222 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 77775.9us
I20260807 11:09:58.730293 27458 raft_consensus.cc:1050] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:09:58.731756 27462 ts_tablet_manager.cc:1918] T adab2949574e4c6badfcf68e51b9c072 P 4fd6381544574ddca559ce63b9243efc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 11:09:58.732529 27462 log.cc:1196] T adab2949574e4c6badfcf68e51b9c072 P 4fd6381544574ddca559ce63b9243efc: Deleting WAL directory at /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-1-root/wals/adab2949574e4c6badfcf68e51b9c072
I20260807 11:09:58.732976 27458 consensus_queue.cc:237] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:58.740532 26984 raft_consensus.cc:1261] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:58.743109 25926 raft_consensus.cc:1261] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:58.746337 27434 consensus_queue.cc:1035] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:58.748196 27458 consensus_queue.cc:1035] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:09:58.782172 27434 raft_consensus.cc:2932] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:58.785205 26984 raft_consensus.cc:2932] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:58.825707 25659 catalog_manager.cc:5495] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:58.837306 26984 raft_consensus.cc:1261] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:58.837608 25917 raft_consensus.cc:1261] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:58.839131 27401 consensus_queue.cc:1035] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:09:58.840849 27461 consensus_queue.cc:1035] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:09:58.837522 25926 raft_consensus.cc:2932] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:58.899927 25800 raft_consensus.cc:2932] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:58.906631 25801 raft_consensus.cc:1261] T 2f60eb5499d64c89b971928b13bcd1be P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:59.129390 26035 raft_consensus.cc:1261] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:09:59.048293 25637 catalog_manager.cc:4841] TS 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): tablet adab2949574e4c6badfcf68e51b9c072 (table test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]) successfully deleted
W20260807 11:09:59.136277 25637 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 88020.5us
I20260807 11:09:59.145227 27464 consensus_queue.cc:1035] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:09:59.198840 27406 consensus_queue.cc:1035] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20260807 11:09:59.269760 27486 raft_consensus.cc:2932] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:59.286439 26034 raft_consensus.cc:2932] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:59.299960 25926 raft_consensus.cc:2932] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:59.312196 26984 raft_consensus.cc:2932] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:59.419077 25660 catalog_manager.cc:5495] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:09:59.439154 25800 raft_consensus.cc:2932] T 2f60eb5499d64c89b971928b13bcd1be P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
W20260807 11:09:59.444378 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 166443us
I20260807 11:09:59.466861 25654 catalog_manager.cc:5495] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:09:59.475847 26034 consensus_queue.cc:237] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:59.523509 25800 consensus_queue.cc:237] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:59.513875 26035 consensus_queue.cc:237] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:09:59.551918 25917 raft_consensus.cc:1261] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:59.553282 26984 raft_consensus.cc:1261] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:59.554239 27395 consensus_queue.cc:1035] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:09:59.558207 27470 consensus_queue.cc:1035] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260807 11:09:59.568887 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 73599.4us
I20260807 11:09:59.573022 25800 raft_consensus.cc:1261] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:59.575523 26986 raft_consensus.cc:1261] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:59.570354 27395 raft_consensus.cc:2932] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:59.583626 27437 consensus_queue.cc:1035] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:09:59.582945 26985 raft_consensus.cc:2932] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:59.589879 27437 consensus_queue.cc:1035] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:09:59.593509 26984 raft_consensus.cc:1261] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:59.597357 27437 raft_consensus.cc:2932] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:59.602540 27504 raft_consensus.cc:2932] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:59.606739 25926 raft_consensus.cc:1261] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:09:59.607862 27458 consensus_queue.cc:1035] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:09:59.609966 25926 raft_consensus.cc:2932] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:59.615520 25639 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 2f60eb5499d64c89b971928b13bcd1be with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 11:09:59.627573 26986 raft_consensus.cc:2932] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:59.631662 25655 catalog_manager.cc:5495] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:59.639002 25637 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 998ef34204b64d2481698cc29e055cc2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 11:09:59.613848 26984 raft_consensus.cc:2932] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:59.623222 27458 consensus_queue.cc:1035] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20260807 11:09:59.616379 25796 raft_consensus.cc:2932] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:59.673090 25639 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet b1765495856a47f09ddbb303f76c635f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 11:09:59.667447 25658 catalog_manager.cc:5495] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 reported cstate change: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:09:59.688833 26032 tablet_service.cc:1514] Processing DeleteTablet for tablet 998ef34204b64d2481698cc29e055cc2 with delete_type TABLET_DATA_TOMBSTONED (TS b22d69db45194908ad3d345eba9badd9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59300
I20260807 11:09:59.688679 25917 raft_consensus.cc:2932] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:59.709805 27507 tablet_replica.cc:330] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9: stopping tablet replica
I20260807 11:09:59.716588 25785 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f60eb5499d64c89b971928b13bcd1be with delete_type TABLET_DATA_TOMBSTONED (TS 606d49e053a346e486f48cb31832b815 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53306
I20260807 11:09:59.717734 27507 raft_consensus.cc:2229] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:59.719280 27507 raft_consensus.cc:2258] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:59.729940 27507 ts_tablet_manager.cc:1905] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 11:09:59.743708 27509 tablet_replica.cc:330] T 2f60eb5499d64c89b971928b13bcd1be P 606d49e053a346e486f48cb31832b815: stopping tablet replica
I20260807 11:09:59.756533 27509 raft_consensus.cc:2229] T 2f60eb5499d64c89b971928b13bcd1be P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:59.757221 27509 raft_consensus.cc:2258] T 2f60eb5499d64c89b971928b13bcd1be P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:59.761787 27509 ts_tablet_manager.cc:1905] T 2f60eb5499d64c89b971928b13bcd1be P 606d49e053a346e486f48cb31832b815: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 11:09:59.757997 25655 catalog_manager.cc:5495] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:09:59.769603 27507 ts_tablet_manager.cc:1918] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 11:09:59.770147 27507 log.cc:1196] T 998ef34204b64d2481698cc29e055cc2 P b22d69db45194908ad3d345eba9badd9: Deleting WAL directory at /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-2-root/wals/998ef34204b64d2481698cc29e055cc2
I20260807 11:09:59.775553 25639 catalog_manager.cc:4841] TS b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): tablet 998ef34204b64d2481698cc29e055cc2 (table test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]) successfully deleted
I20260807 11:09:59.780642 25910 tablet_service.cc:1514] Processing DeleteTablet for tablet b1765495856a47f09ddbb303f76c635f with delete_type TABLET_DATA_TOMBSTONED (TS 4fd6381544574ddca559ce63b9243efc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39730
I20260807 11:09:59.782140 27509 ts_tablet_manager.cc:1918] T 2f60eb5499d64c89b971928b13bcd1be P 606d49e053a346e486f48cb31832b815: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 11:09:59.782711 27509 log.cc:1196] T 2f60eb5499d64c89b971928b13bcd1be P 606d49e053a346e486f48cb31832b815: Deleting WAL directory at /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-0-root/wals/2f60eb5499d64c89b971928b13bcd1be
I20260807 11:09:59.795439 25637 catalog_manager.cc:4841] TS 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): tablet 2f60eb5499d64c89b971928b13bcd1be (table test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]) successfully deleted
I20260807 11:09:59.796482 27510 tablet_replica.cc:330] T b1765495856a47f09ddbb303f76c635f P 4fd6381544574ddca559ce63b9243efc: stopping tablet replica
I20260807 11:09:59.797621 27510 raft_consensus.cc:2229] T b1765495856a47f09ddbb303f76c635f P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:09:59.798426 27510 raft_consensus.cc:2258] T b1765495856a47f09ddbb303f76c635f P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:09:59.801223 27510 ts_tablet_manager.cc:1905] T b1765495856a47f09ddbb303f76c635f P 4fd6381544574ddca559ce63b9243efc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 11:09:59.818485 27510 ts_tablet_manager.cc:1918] T b1765495856a47f09ddbb303f76c635f P 4fd6381544574ddca559ce63b9243efc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 11:09:59.819005 27510 log.cc:1196] T b1765495856a47f09ddbb303f76c635f P 4fd6381544574ddca559ce63b9243efc: Deleting WAL directory at /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-1-root/wals/b1765495856a47f09ddbb303f76c635f
I20260807 11:09:59.821409 25637 catalog_manager.cc:4841] TS 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): tablet b1765495856a47f09ddbb303f76c635f (table test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]) successfully deleted
W20260807 11:09:59.888911 25847 debug-util.cc:398] Leaking SignalData structure 0x7b08000db2a0 after lost signal to thread 24038
W20260807 11:09:59.889847 25847 debug-util.cc:398] Leaking SignalData structure 0x7b080004ea80 after lost signal to thread 25705
W20260807 11:09:59.890743 25847 debug-util.cc:398] Leaking SignalData structure 0x7b08001cab00 after lost signal to thread 25850
W20260807 11:09:59.891539 25847 debug-util.cc:398] Leaking SignalData structure 0x7b080012cf20 after lost signal to thread 25964
W20260807 11:09:59.892309 25847 debug-util.cc:398] Leaking SignalData structure 0x7b0800251740 after lost signal to thread 26083
W20260807 11:09:59.893105 25847 debug-util.cc:398] Leaking SignalData structure 0x7b080032dda0 after lost signal to thread 27031
I20260807 11:10:00.795236 26036 consensus_queue.cc:237] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
W20260807 11:10:00.804075 25981 consensus_peers.cc:487] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 674c5d3caeaf4ccabb23397e655b0e0a. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:10:00.805779 25926 raft_consensus.cc:1261] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:10:00.806967 27504 consensus_queue.cc:1035] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:10:00.807370 25800 raft_consensus.cc:1261] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:10:00.808737 27504 consensus_queue.cc:1035] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:10:00.820112 27406 raft_consensus.cc:2932] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:00.821651 25800 raft_consensus.cc:2932] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:00.871042 25926 raft_consensus.cc:2932] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:00.888356 25655 catalog_manager.cc:5495] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 11:10:00.906409 25918 consensus_queue.cc:237] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
W20260807 11:10:00.919651 25869 consensus_peers.cc:487] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f75da3017fd744719647332808daba49. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:10:00.920094 26036 raft_consensus.cc:1261] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:10:00.921581 27490 consensus_queue.cc:1035] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:10:00.928958 25786 raft_consensus.cc:1261] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:10:00.930552 27490 consensus_queue.cc:1035] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:00.937786 27520 raft_consensus.cc:2932] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:00.955107 25786 raft_consensus.cc:2932] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:00.957383 26036 raft_consensus.cc:2932] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:00.972681 25800 consensus_queue.cc:237] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:00.985138 26036 raft_consensus.cc:1261] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:10:00.985814 25918 raft_consensus.cc:1261] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:10:00.986919 27486 consensus_queue.cc:1035] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:10:00.984937 25654 catalog_manager.cc:5495] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9 reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
W20260807 11:10:00.988222 25750 consensus_peers.cc:487] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51fb619c2ada475f817fbcb9739a401a. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:10:00.988698 27486 consensus_queue.cc:1035] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:10:01.005667 27489 raft_consensus.cc:2932] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:01.007085 25918 raft_consensus.cc:2932] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:01.019623 26036 raft_consensus.cc:2932] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:01.038645 26034 consensus_queue.cc:237] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:01.042398 25654 catalog_manager.cc:5495] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9 reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:01.096822 25918 raft_consensus.cc:1261] T 437f2acbd46c43b2834bf00f08d22f65 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 11:10:01.097124 25981 consensus_peers.cc:487] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 437f2acbd46c43b2834bf00f08d22f65. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:10:01.098685 25800 raft_consensus.cc:1261] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:10:01.099139 27406 consensus_queue.cc:1035] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:10:01.100538 27504 consensus_queue.cc:1035] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:10:01.107827 27460 raft_consensus.cc:2932] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:01.111367 25800 raft_consensus.cc:2932] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:01.117537 25918 raft_consensus.cc:2932] T 437f2acbd46c43b2834bf00f08d22f65 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:01.132436 25654 catalog_manager.cc:5495] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:01.314872 27543 ts_tablet_manager.cc:927] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:01.319608 27543 tablet_copy_client.cc:316] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.130:36465
I20260807 11:10:01.321075 25945 tablet_copy_service.cc:140] P 4fd6381544574ddca559ce63b9243efc: Received BeginTabletCopySession request for tablet f75da3017fd744719647332808daba49 from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:35276)
I20260807 11:10:01.321611 25945 tablet_copy_service.cc:161] P 4fd6381544574ddca559ce63b9243efc: Beginning new tablet copy session on tablet f75da3017fd744719647332808daba49 from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:35276: session id = eb1fca62dcb34ed9b32188ccb19f0543-f75da3017fd744719647332808daba49
I20260807 11:10:01.334906 25945 tablet_copy_source_session.cc:205] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:10:01.339519 27543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75da3017fd744719647332808daba49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:10:01.356228 27545 ts_tablet_manager.cc:927] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:01.357646 27543 tablet_copy_client.cc:791] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:10:01.358399 27543 tablet_copy_client.cc:655] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:10:01.362416 27543 tablet_copy_client.cc:531] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:10:01.371304 27545 tablet_copy_client.cc:316] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.131:46009
I20260807 11:10:01.373838 26061 tablet_copy_service.cc:140] P b22d69db45194908ad3d345eba9badd9: Received BeginTabletCopySession request for tablet 674c5d3caeaf4ccabb23397e655b0e0a from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:43246)
I20260807 11:10:01.374425 26061 tablet_copy_service.cc:161] P b22d69db45194908ad3d345eba9badd9: Beginning new tablet copy session on tablet 674c5d3caeaf4ccabb23397e655b0e0a from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:43246: session id = eb1fca62dcb34ed9b32188ccb19f0543-674c5d3caeaf4ccabb23397e655b0e0a
I20260807 11:10:01.383076 26061 tablet_copy_source_session.cc:205] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:10:01.384915 27543 tablet_bootstrap.cc:492] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
I20260807 11:10:01.394001 27545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674c5d3caeaf4ccabb23397e655b0e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:10:01.417613 27545 tablet_copy_client.cc:791] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:10:01.418427 27545 tablet_copy_client.cc:655] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:10:01.425695 27545 tablet_copy_client.cc:531] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:10:01.428260 27543 tablet_bootstrap.cc:492] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:10:01.429378 27543 tablet_bootstrap.cc:492] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:10:01.430244 27543 ts_tablet_manager.cc:1397] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 0.046s user 0.022s sys 0.008s
I20260807 11:10:01.433462 27543 raft_consensus.cc:348] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:01.434532 27543 raft_consensus.cc:729] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:10:01.435235 27543 consensus_queue.cc:260] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:01.468030 27543 ts_tablet_manager.cc:1428] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.037s user 0.015s sys 0.012s
I20260807 11:10:01.471201 25945 tablet_copy_service.cc:342] P 4fd6381544574ddca559ce63b9243efc: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-f75da3017fd744719647332808daba49 received from {username='slave'} at 127.0.0.1:35276
I20260807 11:10:01.471608 25945 tablet_copy_service.cc:433] P 4fd6381544574ddca559ce63b9243efc: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-f75da3017fd744719647332808daba49 on tablet f75da3017fd744719647332808daba49 with peer eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:01.473246 27545 tablet_bootstrap.cc:492] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
I20260807 11:10:01.479703 27543 ts_tablet_manager.cc:927] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195)
I20260807 11:10:01.512754 27550 ts_tablet_manager.cc:927] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:01.513787 27543 tablet_copy_client.cc:316] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.129:40195
I20260807 11:10:01.517977 25825 tablet_copy_service.cc:140] P 606d49e053a346e486f48cb31832b815: Received BeginTabletCopySession request for tablet 51fb619c2ada475f817fbcb9739a401a from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:59118)
I20260807 11:10:01.518679 25825 tablet_copy_service.cc:161] P 606d49e053a346e486f48cb31832b815: Beginning new tablet copy session on tablet 51fb619c2ada475f817fbcb9739a401a from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:59118: session id = eb1fca62dcb34ed9b32188ccb19f0543-51fb619c2ada475f817fbcb9739a401a
I20260807 11:10:01.534734 25825 tablet_copy_source_session.cc:205] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:10:01.539291 27543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51fb619c2ada475f817fbcb9739a401a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:10:01.548075 27545 tablet_bootstrap.cc:492] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:10:01.549422 27545 tablet_bootstrap.cc:492] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:10:01.550390 27545 ts_tablet_manager.cc:1397] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 0.077s user 0.015s sys 0.022s
I20260807 11:10:01.555485 27545 raft_consensus.cc:348] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:01.556849 27545 raft_consensus.cc:729] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:10:01.557716 27545 consensus_queue.cc:260] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:01.561419 27545 ts_tablet_manager.cc:1428] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.011s user 0.004s sys 0.005s
I20260807 11:10:01.563308 26061 tablet_copy_service.cc:342] P b22d69db45194908ad3d345eba9badd9: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-674c5d3caeaf4ccabb23397e655b0e0a received from {username='slave'} at 127.0.0.1:43246
I20260807 11:10:01.563896 26061 tablet_copy_service.cc:433] P b22d69db45194908ad3d345eba9badd9: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-674c5d3caeaf4ccabb23397e655b0e0a on tablet 674c5d3caeaf4ccabb23397e655b0e0a with peer eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:01.565682 27550 tablet_copy_client.cc:316] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.131:46009
I20260807 11:10:01.569381 26061 tablet_copy_service.cc:140] P b22d69db45194908ad3d345eba9badd9: Received BeginTabletCopySession request for tablet 437f2acbd46c43b2834bf00f08d22f65 from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:43246)
I20260807 11:10:01.570181 26061 tablet_copy_service.cc:161] P b22d69db45194908ad3d345eba9badd9: Beginning new tablet copy session on tablet 437f2acbd46c43b2834bf00f08d22f65 from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:43246: session id = eb1fca62dcb34ed9b32188ccb19f0543-437f2acbd46c43b2834bf00f08d22f65
I20260807 11:10:01.575625 27543 tablet_copy_client.cc:791] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:10:01.576396 27543 tablet_copy_client.cc:655] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:10:01.582214 27543 tablet_copy_client.cc:531] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:10:01.592836 27543 tablet_bootstrap.cc:492] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
I20260807 11:10:01.597640 26061 tablet_copy_source_session.cc:205] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:10:01.601926 27550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 437f2acbd46c43b2834bf00f08d22f65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:10:01.675854 27550 tablet_copy_client.cc:791] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:10:01.676852 27550 tablet_copy_client.cc:655] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:10:01.685738 27543 tablet_bootstrap.cc:492] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:10:01.687131 27543 tablet_bootstrap.cc:492] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:10:01.689036 27543 ts_tablet_manager.cc:1397] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 0.096s user 0.030s sys 0.038s
I20260807 11:10:01.691941 27550 tablet_copy_client.cc:531] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:10:01.692931 27543 raft_consensus.cc:348] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:01.694350 27543 raft_consensus.cc:729] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:10:01.695401 27543 consensus_queue.cc:260] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:01.707703 27543 ts_tablet_manager.cc:1428] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.018s user 0.008s sys 0.000s
I20260807 11:10:01.714159 25825 tablet_copy_service.cc:342] P 606d49e053a346e486f48cb31832b815: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-51fb619c2ada475f817fbcb9739a401a received from {username='slave'} at 127.0.0.1:59118
I20260807 11:10:01.714573 25825 tablet_copy_service.cc:433] P 606d49e053a346e486f48cb31832b815: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-51fb619c2ada475f817fbcb9739a401a on tablet 51fb619c2ada475f817fbcb9739a401a with peer eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:01.785192 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 53317.7us
I20260807 11:10:01.793495 26986 raft_consensus.cc:1203] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:10:01.835350 27550 tablet_bootstrap.cc:492] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
W20260807 11:10:01.835861 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 49982.2us
I20260807 11:10:01.883329 27550 tablet_bootstrap.cc:492] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:10:01.886029 27550 tablet_bootstrap.cc:492] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:10:01.890677 26986 raft_consensus.cc:1203] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:10:01.894404 27550 ts_tablet_manager.cc:1397] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 0.059s user 0.026s sys 0.013s
I20260807 11:10:01.900120 27550 raft_consensus.cc:348] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:01.901784 27550 raft_consensus.cc:729] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:10:01.904943 27550 consensus_queue.cc:260] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:01.915890 27550 ts_tablet_manager.cc:1428] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.021s user 0.011s sys 0.000s
I20260807 11:10:01.918260 26061 tablet_copy_service.cc:342] P b22d69db45194908ad3d345eba9badd9: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-437f2acbd46c43b2834bf00f08d22f65 received from {username='slave'} at 127.0.0.1:43246
I20260807 11:10:01.918812 26061 tablet_copy_service.cc:433] P b22d69db45194908ad3d345eba9badd9: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-437f2acbd46c43b2834bf00f08d22f65 on tablet 437f2acbd46c43b2834bf00f08d22f65 with peer eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:02.089589 26986 raft_consensus.cc:1203] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:10:02.201731 27542 raft_consensus.cc:1050] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:10:02.204382 27542 consensus_queue.cc:237] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:02.210796 26986 raft_consensus.cc:1261] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:02.212293 27558 consensus_queue.cc:1035] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:10:02.214295 25926 raft_consensus.cc:1261] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:02.215848 25786 raft_consensus.cc:1261] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:02.216130 27558 consensus_queue.cc:1035] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:10:02.222577 27558 consensus_queue.cc:1035] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:10:02.228681 27541 raft_consensus.cc:2932] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.231426 25917 raft_consensus.cc:2932] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.232491 25786 raft_consensus.cc:2932] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.242704 26986 raft_consensus.cc:2932] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.255824 25655 catalog_manager.cc:5495] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.292652 26041 consensus_queue.cc:237] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:02.298296 26984 raft_consensus.cc:1261] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:10:02.309437 25918 raft_consensus.cc:1261] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:10:02.311928 27558 consensus_queue.cc:1035] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:10:02.327039 27542 consensus_queue.cc:1035] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:10:02.349180 27541 raft_consensus.cc:2932] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.354911 25920 raft_consensus.cc:2932] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.356618 26984 raft_consensus.cc:2932] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.363605 26986 raft_consensus.cc:1203] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:10:02.368010 27409 raft_consensus.cc:1050] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:10:02.370330 27409 consensus_queue.cc:237] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:02.380206 25639 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 674c5d3caeaf4ccabb23397e655b0e0a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 11:10:02.387363 25658 catalog_manager.cc:5495] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.397779 26036 raft_consensus.cc:1261] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:02.401448 25785 tablet_service.cc:1514] Processing DeleteTablet for tablet 674c5d3caeaf4ccabb23397e655b0e0a with delete_type TABLET_DATA_TOMBSTONED (TS 606d49e053a346e486f48cb31832b815 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53306
I20260807 11:10:02.404430 26986 raft_consensus.cc:1261] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:02.402956 27520 consensus_queue.cc:1035] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260807 11:10:02.414000 27520 consensus_queue.cc:1035] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:10:02.420173 27580 tablet_replica.cc:330] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815: stopping tablet replica
I20260807 11:10:02.429378 27580 raft_consensus.cc:2229] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:10:02.430645 27580 raft_consensus.cc:2258] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:10:02.435956 25786 raft_consensus.cc:1261] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:02.438230 27583 consensus_queue.cc:1035] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:10:02.439644 27580 ts_tablet_manager.cc:1905] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 11:10:02.461180 27580 ts_tablet_manager.cc:1918] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 11:10:02.460067 27583 raft_consensus.cc:2932] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.461736 27580 log.cc:1196] T 674c5d3caeaf4ccabb23397e655b0e0a P 606d49e053a346e486f48cb31832b815: Deleting WAL directory at /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-0-root/wals/674c5d3caeaf4ccabb23397e655b0e0a
I20260807 11:10:02.463795 25796 raft_consensus.cc:2932] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.470635 25637 catalog_manager.cc:4841] TS 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): tablet 674c5d3caeaf4ccabb23397e655b0e0a (table test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]) successfully deleted
I20260807 11:10:02.470069 26036 raft_consensus.cc:2932] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.474558 26986 raft_consensus.cc:2932] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.487385 25658 catalog_manager.cc:5495] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.509418 25920 consensus_queue.cc:237] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:02.512004 27559 raft_consensus.cc:1050] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:10:02.514369 27559 consensus_queue.cc:237] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:02.523015 26985 raft_consensus.cc:1261] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:02.523515 26986 raft_consensus.cc:1261] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:10:02.523890 26036 raft_consensus.cc:1261] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:02.525336 27586 consensus_queue.cc:1035] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:10:02.525604 25920 raft_consensus.cc:1261] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:02.525977 27533 consensus_queue.cc:1035] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:10:02.528239 27533 consensus_queue.cc:1035] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:10:02.530915 27489 consensus_queue.cc:1035] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:10:02.533003 25796 raft_consensus.cc:1261] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:10:02.534291 27556 consensus_queue.cc:1035] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:10:02.538445 27520 raft_consensus.cc:2932] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.548417 25637 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet f75da3017fd744719647332808daba49 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 11:10:02.552035 25800 raft_consensus.cc:2932] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.560001 26985 raft_consensus.cc:2932] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.571718 25658 catalog_manager.cc:5495] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 reported cstate change: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.582530 27489 raft_consensus.cc:2932] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.584662 26032 tablet_service.cc:1514] Processing DeleteTablet for tablet f75da3017fd744719647332808daba49 with delete_type TABLET_DATA_TOMBSTONED (TS b22d69db45194908ad3d345eba9badd9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59300
I20260807 11:10:02.584533 25917 raft_consensus.cc:2932] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.597508 27594 tablet_replica.cc:330] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9: stopping tablet replica
I20260807 11:10:02.610646 26985 raft_consensus.cc:2932] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.613173 26034 raft_consensus.cc:2932] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.615440 27594 raft_consensus.cc:2229] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:10:02.616534 27594 raft_consensus.cc:2258] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:10:02.621372 27594 ts_tablet_manager.cc:1905] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 11:10:02.639840 25658 catalog_manager.cc:5495] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.649123 27594 ts_tablet_manager.cc:1918] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 11:10:02.649688 27594 log.cc:1196] T f75da3017fd744719647332808daba49 P b22d69db45194908ad3d345eba9badd9: Deleting WAL directory at /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-2-root/wals/f75da3017fd744719647332808daba49
I20260807 11:10:02.651855 25639 catalog_manager.cc:4841] TS b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): tablet f75da3017fd744719647332808daba49 (table test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]) successfully deleted
I20260807 11:10:02.674250 25796 consensus_queue.cc:237] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:02.681293 26985 raft_consensus.cc:1261] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:10:02.683164 27559 consensus_queue.cc:1035] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:10:02.687551 25926 raft_consensus.cc:1261] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:10:02.692251 27489 consensus_queue.cc:1035] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:10:02.699537 27486 raft_consensus.cc:2932] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.702461 26985 raft_consensus.cc:2932] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.705088 25920 raft_consensus.cc:2932] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.716753 25637 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 51fb619c2ada475f817fbcb9739a401a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 11:10:02.730572 25655 catalog_manager.cc:5495] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 reported cstate change: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:10:02.750799 26032 tablet_service.cc:1514] Processing DeleteTablet for tablet 51fb619c2ada475f817fbcb9739a401a with delete_type TABLET_DATA_TOMBSTONED (TS b22d69db45194908ad3d345eba9badd9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59300
I20260807 11:10:02.752008 27594 tablet_replica.cc:330] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9: stopping tablet replica
I20260807 11:10:02.756488 27594 raft_consensus.cc:2229] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:10:02.757601 27594 raft_consensus.cc:2258] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:10:02.769894 27594 ts_tablet_manager.cc:1905] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 11:10:02.794673 27594 ts_tablet_manager.cc:1918] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 11:10:02.795145 27594 log.cc:1196] T 51fb619c2ada475f817fbcb9739a401a P b22d69db45194908ad3d345eba9badd9: Deleting WAL directory at /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-2-root/wals/51fb619c2ada475f817fbcb9739a401a
I20260807 11:10:02.796882 25639 catalog_manager.cc:4841] TS b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): tablet 51fb619c2ada475f817fbcb9739a401a (table test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]) successfully deleted
I20260807 11:10:02.846145 27541 raft_consensus.cc:1050] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:10:02.848543 27541 consensus_queue.cc:237] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:02.882311 26986 raft_consensus.cc:1261] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:02.883148 25920 raft_consensus.cc:1261] T 437f2acbd46c43b2834bf00f08d22f65 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:02.883685 27541 consensus_queue.cc:1035] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:10:02.884372 25800 raft_consensus.cc:1261] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:02.885262 27558 consensus_queue.cc:1035] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:10:02.888458 27575 consensus_queue.cc:1035] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:10:02.921111 27575 raft_consensus.cc:2932] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.923995 26986 raft_consensus.cc:2932] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.926934 25920 raft_consensus.cc:2932] T 437f2acbd46c43b2834bf00f08d22f65 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.935976 25655 catalog_manager.cc:5495] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.946594 25800 raft_consensus.cc:2932] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:02.974323 26036 consensus_queue.cc:237] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:02.979506 26986 raft_consensus.cc:1261] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:10:02.987792 25800 raft_consensus.cc:1261] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:10:02.988660 27558 consensus_queue.cc:1035] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:10:02.990513 27541 consensus_queue.cc:1035] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:10:02.997995 27541 raft_consensus.cc:2932] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:03.002878 25796 raft_consensus.cc:2932] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:03.008956 25639 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 437f2acbd46c43b2834bf00f08d22f65 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 11:10:03.008333 26986 raft_consensus.cc:2932] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:03.021271 25660 catalog_manager.cc:5495] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 reported cstate change: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:10:03.036695 25910 tablet_service.cc:1514] Processing DeleteTablet for tablet 437f2acbd46c43b2834bf00f08d22f65 with delete_type TABLET_DATA_TOMBSTONED (TS 4fd6381544574ddca559ce63b9243efc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39730
I20260807 11:10:03.038759 27601 tablet_replica.cc:330] T 437f2acbd46c43b2834bf00f08d22f65 P 4fd6381544574ddca559ce63b9243efc: stopping tablet replica
I20260807 11:10:03.039768 27601 raft_consensus.cc:2229] T 437f2acbd46c43b2834bf00f08d22f65 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:10:03.040522 27601 raft_consensus.cc:2258] T 437f2acbd46c43b2834bf00f08d22f65 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:10:03.044006 27601 ts_tablet_manager.cc:1905] T 437f2acbd46c43b2834bf00f08d22f65 P 4fd6381544574ddca559ce63b9243efc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 11:10:03.057704 27601 ts_tablet_manager.cc:1918] T 437f2acbd46c43b2834bf00f08d22f65 P 4fd6381544574ddca559ce63b9243efc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 11:10:03.058110 27601 log.cc:1196] T 437f2acbd46c43b2834bf00f08d22f65 P 4fd6381544574ddca559ce63b9243efc: Deleting WAL directory at /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-1-root/wals/437f2acbd46c43b2834bf00f08d22f65
I20260807 11:10:03.059569 25637 catalog_manager.cc:4841] TS 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): tablet 437f2acbd46c43b2834bf00f08d22f65 (table test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]) successfully deleted
I20260807 11:10:04.105779 26041 consensus_queue.cc:237] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:04.121079 25800 raft_consensus.cc:1261] T 54a291aa09434b06adbb86586dacec0e P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:10:04.124415 25918 raft_consensus.cc:1261] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:10:04.128293 27608 consensus_queue.cc:1035] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:04.129602 27558 consensus_queue.cc:1035] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260807 11:10:04.134545 25981 consensus_peers.cc:487] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54a291aa09434b06adbb86586dacec0e. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:10:04.148857 27575 raft_consensus.cc:2932] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:04.153803 25796 raft_consensus.cc:2932] T 54a291aa09434b06adbb86586dacec0e P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:04.157645 25920 raft_consensus.cc:2932] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:04.165158 25800 consensus_queue.cc:237] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:04.175523 25660 catalog_manager.cc:5495] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 11:10:04.186595 25920 raft_consensus.cc:1261] T d8012276091e4b63b062cb7152208a16 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 11:10:04.187552 25750 consensus_peers.cc:487] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d8012276091e4b63b062cb7152208a16. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:10:04.187963 26036 raft_consensus.cc:1261] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:10:04.188738 27533 consensus_queue.cc:1035] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:04.190030 27486 consensus_queue.cc:1035] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:04.219148 27603 raft_consensus.cc:2932] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:04.231103 25920 raft_consensus.cc:2932] T d8012276091e4b63b062cb7152208a16 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:04.236392 26036 raft_consensus.cc:2932] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:04.245952 25920 consensus_queue.cc:237] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:04.257838 25658 catalog_manager.cc:5495] T d8012276091e4b63b062cb7152208a16 P 4fd6381544574ddca559ce63b9243efc reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
W20260807 11:10:04.289093 25869 consensus_peers.cc:487] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c69ebea3aeb47d48ab8e083fdf71f7c. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:10:04.292853 26041 raft_consensus.cc:1261] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:10:04.294265 27556 consensus_queue.cc:1035] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:04.295423 25800 raft_consensus.cc:1261] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:10:04.296836 27520 consensus_queue.cc:1035] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:04.306944 27556 raft_consensus.cc:2932] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:04.308796 26041 raft_consensus.cc:2932] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:04.311122 25800 raft_consensus.cc:2932] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:04.331425 25660 catalog_manager.cc:5495] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:04.341315 25800 consensus_queue.cc:237] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:04.350908 25920 raft_consensus.cc:1261] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:10:04.351198 26041 raft_consensus.cc:1261] T 9c6408e9acee46adacd2bbd81669f27a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 11:10:04.352064 25750 consensus_peers.cc:487] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9c6408e9acee46adacd2bbd81669f27a. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:10:04.353520 27626 consensus_queue.cc:1035] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:04.359941 27626 consensus_queue.cc:1035] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:10:04.369004 27603 raft_consensus.cc:2932] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:04.371716 25920 raft_consensus.cc:2932] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:04.385934 26036 raft_consensus.cc:2932] T 9c6408e9acee46adacd2bbd81669f27a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:04.389359 25655 catalog_manager.cc:5495] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 11:10:04.606338 27639 ts_tablet_manager.cc:927] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195)
I20260807 11:10:04.608464 27639 tablet_copy_client.cc:316] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.129:40195
I20260807 11:10:04.614912 25825 tablet_copy_service.cc:140] P 606d49e053a346e486f48cb31832b815: Received BeginTabletCopySession request for tablet d8012276091e4b63b062cb7152208a16 from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:59118)
I20260807 11:10:04.615466 25825 tablet_copy_service.cc:161] P 606d49e053a346e486f48cb31832b815: Beginning new tablet copy session on tablet d8012276091e4b63b062cb7152208a16 from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:59118: session id = eb1fca62dcb34ed9b32188ccb19f0543-d8012276091e4b63b062cb7152208a16
I20260807 11:10:04.623487 25825 tablet_copy_source_session.cc:205] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:10:04.626765 27639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8012276091e4b63b062cb7152208a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:10:04.640625 27639 tablet_copy_client.cc:791] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:10:04.641291 27639 tablet_copy_client.cc:655] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:10:04.646785 27639 tablet_copy_client.cc:531] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:10:04.659250 27639 tablet_bootstrap.cc:492] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
I20260807 11:10:04.686547 27639 tablet_bootstrap.cc:492] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:10:04.687472 27639 tablet_bootstrap.cc:492] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:10:04.688166 27639 ts_tablet_manager.cc:1397] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 0.029s user 0.014s sys 0.007s
I20260807 11:10:04.691205 27639 raft_consensus.cc:348] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:04.692162 27639 raft_consensus.cc:729] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:10:04.692849 27639 consensus_queue.cc:260] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:04.699994 27642 ts_tablet_manager.cc:927] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:04.707571 27639 ts_tablet_manager.cc:1428] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.019s user 0.012s sys 0.009s
I20260807 11:10:04.710000 25825 tablet_copy_service.cc:342] P 606d49e053a346e486f48cb31832b815: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-d8012276091e4b63b062cb7152208a16 received from {username='slave'} at 127.0.0.1:59118
I20260807 11:10:04.710393 25825 tablet_copy_service.cc:433] P 606d49e053a346e486f48cb31832b815: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-d8012276091e4b63b062cb7152208a16 on tablet d8012276091e4b63b062cb7152208a16 with peer eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:04.711701 27642 tablet_copy_client.cc:316] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.131:46009
I20260807 11:10:04.714316 26061 tablet_copy_service.cc:140] P b22d69db45194908ad3d345eba9badd9: Received BeginTabletCopySession request for tablet 54a291aa09434b06adbb86586dacec0e from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:43246)
I20260807 11:10:04.714850 26061 tablet_copy_service.cc:161] P b22d69db45194908ad3d345eba9badd9: Beginning new tablet copy session on tablet 54a291aa09434b06adbb86586dacec0e from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:43246: session id = eb1fca62dcb34ed9b32188ccb19f0543-54a291aa09434b06adbb86586dacec0e
I20260807 11:10:04.726087 26061 tablet_copy_source_session.cc:205] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:10:04.731001 27642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54a291aa09434b06adbb86586dacec0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:10:04.745908 27642 tablet_copy_client.cc:791] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:10:04.746548 27642 tablet_copy_client.cc:655] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:10:04.751318 27642 tablet_copy_client.cc:531] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:10:04.761941 27642 tablet_bootstrap.cc:492] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
I20260807 11:10:04.762329 27639 ts_tablet_manager.cc:927] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195)
I20260807 11:10:04.786552 27639 tablet_copy_client.cc:316] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.129:40195
I20260807 11:10:04.790661 25825 tablet_copy_service.cc:140] P 606d49e053a346e486f48cb31832b815: Received BeginTabletCopySession request for tablet 9c6408e9acee46adacd2bbd81669f27a from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:59118)
I20260807 11:10:04.792203 25825 tablet_copy_service.cc:161] P 606d49e053a346e486f48cb31832b815: Beginning new tablet copy session on tablet 9c6408e9acee46adacd2bbd81669f27a from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:59118: session id = eb1fca62dcb34ed9b32188ccb19f0543-9c6408e9acee46adacd2bbd81669f27a
I20260807 11:10:04.805755 25825 tablet_copy_source_session.cc:205] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:10:04.805977 27642 tablet_bootstrap.cc:492] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:10:04.808422 27642 tablet_bootstrap.cc:492] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:10:04.809430 27642 ts_tablet_manager.cc:1397] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 0.048s user 0.023s sys 0.015s
I20260807 11:10:04.811647 27639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c6408e9acee46adacd2bbd81669f27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:10:04.814781 27642 raft_consensus.cc:348] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:04.816165 27642 raft_consensus.cc:729] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:10:04.817286 27642 consensus_queue.cc:260] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:04.827503 27642 ts_tablet_manager.cc:1428] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.018s user 0.004s sys 0.008s
I20260807 11:10:04.833032 26061 tablet_copy_service.cc:342] P b22d69db45194908ad3d345eba9badd9: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-54a291aa09434b06adbb86586dacec0e received from {username='slave'} at 127.0.0.1:43246
I20260807 11:10:04.833590 26061 tablet_copy_service.cc:433] P b22d69db45194908ad3d345eba9badd9: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-54a291aa09434b06adbb86586dacec0e on tablet 54a291aa09434b06adbb86586dacec0e with peer eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:04.850271 27639 tablet_copy_client.cc:791] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:10:04.850951 27639 tablet_copy_client.cc:655] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:10:04.855669 27639 tablet_copy_client.cc:531] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:10:04.873168 27639 tablet_bootstrap.cc:492] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
I20260807 11:10:04.892508 27639 tablet_bootstrap.cc:492] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:10:04.893399 27639 tablet_bootstrap.cc:492] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:10:04.894050 27639 ts_tablet_manager.cc:1397] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.009s
I20260807 11:10:04.896641 27639 raft_consensus.cc:348] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:04.897548 27639 raft_consensus.cc:729] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:10:04.898250 27639 consensus_queue.cc:260] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:04.901365 27639 ts_tablet_manager.cc:1428] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260807 11:10:04.902945 25825 tablet_copy_service.cc:342] P 606d49e053a346e486f48cb31832b815: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-9c6408e9acee46adacd2bbd81669f27a received from {username='slave'} at 127.0.0.1:59118
I20260807 11:10:04.903345 25825 tablet_copy_service.cc:433] P 606d49e053a346e486f48cb31832b815: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-9c6408e9acee46adacd2bbd81669f27a on tablet 9c6408e9acee46adacd2bbd81669f27a with peer eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:04.903578 27642 ts_tablet_manager.cc:927] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:04.905197 27642 tablet_copy_client.cc:316] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.130:36465
I20260807 11:10:04.907054 25945 tablet_copy_service.cc:140] P 4fd6381544574ddca559ce63b9243efc: Received BeginTabletCopySession request for tablet 3c69ebea3aeb47d48ab8e083fdf71f7c from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:35276)
I20260807 11:10:04.907603 25945 tablet_copy_service.cc:161] P 4fd6381544574ddca559ce63b9243efc: Beginning new tablet copy session on tablet 3c69ebea3aeb47d48ab8e083fdf71f7c from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:35276: session id = eb1fca62dcb34ed9b32188ccb19f0543-3c69ebea3aeb47d48ab8e083fdf71f7c
I20260807 11:10:04.916400 25945 tablet_copy_source_session.cc:205] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:10:04.919513 27642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c69ebea3aeb47d48ab8e083fdf71f7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:10:04.935819 27642 tablet_copy_client.cc:791] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:10:04.936384 27642 tablet_copy_client.cc:655] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:10:04.940194 27642 tablet_copy_client.cc:531] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:10:04.951335 27642 tablet_bootstrap.cc:492] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
I20260807 11:10:04.985932 27642 tablet_bootstrap.cc:492] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:10:04.987803 27642 tablet_bootstrap.cc:492] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:10:04.988682 27642 ts_tablet_manager.cc:1397] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 0.038s user 0.014s sys 0.011s
I20260807 11:10:04.992512 27642 raft_consensus.cc:348] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:04.993481 27642 raft_consensus.cc:729] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:10:04.994119 27642 consensus_queue.cc:260] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:04.996996 27642 ts_tablet_manager.cc:1428] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20260807 11:10:05.004146 25945 tablet_copy_service.cc:342] P 4fd6381544574ddca559ce63b9243efc: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-3c69ebea3aeb47d48ab8e083fdf71f7c received from {username='slave'} at 127.0.0.1:35276
I20260807 11:10:05.004650 25945 tablet_copy_service.cc:433] P 4fd6381544574ddca559ce63b9243efc: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-3c69ebea3aeb47d48ab8e083fdf71f7c on tablet 3c69ebea3aeb47d48ab8e083fdf71f7c with peer eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:05.085772 26986 raft_consensus.cc:1203] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:10:05.108134 26986 raft_consensus.cc:1203] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:10:05.148382 26986 raft_consensus.cc:1203] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:10:05.416455 26986 raft_consensus.cc:1203] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:10:05.522396 27648 raft_consensus.cc:1050] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:10:05.524842 27648 consensus_queue.cc:237] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:05.532140 26986 raft_consensus.cc:1261] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:05.533215 25917 raft_consensus.cc:1261] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:05.533614 27575 consensus_queue.cc:1035] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:10:05.535975 27648 consensus_queue.cc:1035] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:10:05.538461 25786 raft_consensus.cc:1261] T 54a291aa09434b06adbb86586dacec0e P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:05.540650 27648 consensus_queue.cc:1035] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260807 11:10:05.551026 27575 raft_consensus.cc:2932] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.553596 25786 raft_consensus.cc:2932] T 54a291aa09434b06adbb86586dacec0e P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.556376 26986 raft_consensus.cc:2932] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.566011 25917 raft_consensus.cc:2932] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.571569 25655 catalog_manager.cc:5495] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.591667 27603 raft_consensus.cc:1050] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:10:05.594302 27603 consensus_queue.cc:237] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:05.602269 26041 consensus_queue.cc:237] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:05.603173 26985 raft_consensus.cc:1261] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:05.604444 26036 raft_consensus.cc:1261] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:05.606148 27486 consensus_queue.cc:1035] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:10:05.606995 27627 consensus_queue.cc:1035] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:10:05.611850 25917 raft_consensus.cc:1261] T d8012276091e4b63b062cb7152208a16 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:05.612049 25918 raft_consensus.cc:1261] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:10:05.613794 27575 consensus_queue.cc:1035] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:10:05.614919 26985 raft_consensus.cc:1261] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:10:05.616005 27603 consensus_queue.cc:1035] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:10:05.616578 27656 consensus_queue.cc:1035] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:10:05.622447 27656 raft_consensus.cc:2932] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.625049 26984 raft_consensus.cc:2932] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.635908 25917 raft_consensus.cc:2932] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.638461 25639 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 54a291aa09434b06adbb86586dacec0e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 11:10:05.660214 25654 catalog_manager.cc:5495] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 reported cstate change: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New cstate: current_term: 1 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:10:05.674365 25785 tablet_service.cc:1514] Processing DeleteTablet for tablet 54a291aa09434b06adbb86586dacec0e with delete_type TABLET_DATA_TOMBSTONED (TS 606d49e053a346e486f48cb31832b815 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53306
I20260807 11:10:05.680467 27685 tablet_replica.cc:330] T 54a291aa09434b06adbb86586dacec0e P 606d49e053a346e486f48cb31832b815: stopping tablet replica
I20260807 11:10:05.692662 27685 raft_consensus.cc:2229] T 54a291aa09434b06adbb86586dacec0e P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:10:05.693672 27685 raft_consensus.cc:2258] T 54a291aa09434b06adbb86586dacec0e P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:10:05.697914 27486 raft_consensus.cc:2932] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.699554 27685 ts_tablet_manager.cc:1905] T 54a291aa09434b06adbb86586dacec0e P 606d49e053a346e486f48cb31832b815: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 11:10:05.702603 26041 raft_consensus.cc:2932] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.709894 27627 raft_consensus.cc:1050] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:10:05.714282 26984 raft_consensus.cc:2932] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.720279 27627 consensus_queue.cc:237] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:05.722352 25654 catalog_manager.cc:5495] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.708490 25918 raft_consensus.cc:2932] T d8012276091e4b63b062cb7152208a16 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.740958 26986 raft_consensus.cc:1261] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:05.742702 25917 raft_consensus.cc:1261] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:05.743605 26041 raft_consensus.cc:1261] T 9c6408e9acee46adacd2bbd81669f27a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:05.743188 27625 consensus_queue.cc:1035] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:10:05.744858 27687 consensus_queue.cc:1035] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:10:05.746253 27603 consensus_queue.cc:1035] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:10:05.761503 27685 ts_tablet_manager.cc:1918] T 54a291aa09434b06adbb86586dacec0e P 606d49e053a346e486f48cb31832b815: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 11:10:05.762130 27685 log.cc:1196] T 54a291aa09434b06adbb86586dacec0e P 606d49e053a346e486f48cb31832b815: Deleting WAL directory at /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-0-root/wals/54a291aa09434b06adbb86586dacec0e
I20260807 11:10:05.766461 25637 catalog_manager.cc:4841] TS 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): tablet 54a291aa09434b06adbb86586dacec0e (table test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]) successfully deleted
I20260807 11:10:05.768477 26986 raft_consensus.cc:2932] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.771363 26036 raft_consensus.cc:2932] T 9c6408e9acee46adacd2bbd81669f27a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.773168 27603 raft_consensus.cc:2932] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.786586 25654 catalog_manager.cc:5495] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.793010 25786 consensus_queue.cc:237] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:05.799309 26041 raft_consensus.cc:1261] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:10:05.799814 26986 raft_consensus.cc:1261] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:10:05.801376 27627 consensus_queue.cc:1035] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:10:05.797564 25918 raft_consensus.cc:2932] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.803961 27625 consensus_queue.cc:1035] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:10:05.817677 27687 raft_consensus.cc:2932] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.825165 25799 consensus_queue.cc:237] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:05.825333 26984 raft_consensus.cc:2932] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.833832 26036 raft_consensus.cc:2932] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.838126 25926 raft_consensus.cc:1261] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:10:05.839046 25637 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d8012276091e4b63b062cb7152208a16 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 11:10:05.843887 27603 consensus_queue.cc:1035] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:10:05.846869 26986 raft_consensus.cc:1261] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:10:05.858222 27687 raft_consensus.cc:2932] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.864470 27625 consensus_queue.cc:1035] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:10:05.872140 25637 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 9c6408e9acee46adacd2bbd81669f27a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 11:10:05.878710 26986 raft_consensus.cc:2932] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.863991 25918 raft_consensus.cc:2932] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.861069 25654 catalog_manager.cc:5495] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 reported cstate change: config changed from index 3 to 4, VOTER 4fd6381544574ddca559ce63b9243efc (127.23.120.130) evicted. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:10:05.913345 25910 tablet_service.cc:1514] Processing DeleteTablet for tablet d8012276091e4b63b062cb7152208a16 with delete_type TABLET_DATA_TOMBSTONED (TS 4fd6381544574ddca559ce63b9243efc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39730
I20260807 11:10:05.911893 25660 catalog_manager.cc:5495] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: config changed from index 3 to 4, VOTER b22d69db45194908ad3d345eba9badd9 (127.23.120.131) evicted. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.917579 27520 raft_consensus.cc:1050] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:10:05.919922 27520 consensus_queue.cc:237] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:05.930548 26032 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c6408e9acee46adacd2bbd81669f27a with delete_type TABLET_DATA_TOMBSTONED (TS b22d69db45194908ad3d345eba9badd9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59300
I20260807 11:10:05.933305 27697 tablet_replica.cc:330] T d8012276091e4b63b062cb7152208a16 P 4fd6381544574ddca559ce63b9243efc: stopping tablet replica
I20260807 11:10:05.934283 27697 raft_consensus.cc:2229] T d8012276091e4b63b062cb7152208a16 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:10:05.940567 26984 raft_consensus.cc:1261] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:05.941641 26041 raft_consensus.cc:1261] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:05.942932 27632 consensus_queue.cc:1035] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:10:05.945815 27556 consensus_queue.cc:1035] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:10:05.947141 27703 tablet_replica.cc:330] T 9c6408e9acee46adacd2bbd81669f27a P b22d69db45194908ad3d345eba9badd9: stopping tablet replica
I20260807 11:10:05.948357 27703 raft_consensus.cc:2229] T 9c6408e9acee46adacd2bbd81669f27a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:10:05.949190 27703 raft_consensus.cc:2258] T 9c6408e9acee46adacd2bbd81669f27a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:10:05.954573 27703 ts_tablet_manager.cc:1905] T 9c6408e9acee46adacd2bbd81669f27a P b22d69db45194908ad3d345eba9badd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 11:10:05.957454 25796 raft_consensus.cc:1261] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:05.935019 27697 raft_consensus.cc:2258] T d8012276091e4b63b062cb7152208a16 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:10:05.958726 27556 consensus_queue.cc:1035] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:10:05.963472 27697 ts_tablet_manager.cc:1905] T d8012276091e4b63b062cb7152208a16 P 4fd6381544574ddca559ce63b9243efc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 11:10:05.979323 27703 ts_tablet_manager.cc:1918] T 9c6408e9acee46adacd2bbd81669f27a P b22d69db45194908ad3d345eba9badd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 11:10:05.978780 27520 raft_consensus.cc:2932] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.979925 27703 log.cc:1196] T 9c6408e9acee46adacd2bbd81669f27a P b22d69db45194908ad3d345eba9badd9: Deleting WAL directory at /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-2-root/wals/9c6408e9acee46adacd2bbd81669f27a
I20260807 11:10:05.979653 25796 raft_consensus.cc:2932] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.983273 25639 catalog_manager.cc:4841] TS b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): tablet 9c6408e9acee46adacd2bbd81669f27a (table test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]) successfully deleted
I20260807 11:10:05.994498 26984 raft_consensus.cc:2932] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:05.999169 27697 ts_tablet_manager.cc:1918] T d8012276091e4b63b062cb7152208a16 P 4fd6381544574ddca559ce63b9243efc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 11:10:05.999773 27697 log.cc:1196] T d8012276091e4b63b062cb7152208a16 P 4fd6381544574ddca559ce63b9243efc: Deleting WAL directory at /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-1-root/wals/d8012276091e4b63b062cb7152208a16
I20260807 11:10:06.002851 25654 catalog_manager.cc:5495] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 606d49e053a346e486f48cb31832b815 reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:06.010833 25637 catalog_manager.cc:4841] TS 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): tablet d8012276091e4b63b062cb7152208a16 (table test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]) successfully deleted
I20260807 11:10:06.013089 26041 raft_consensus.cc:2932] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:06.035349 25920 consensus_queue.cc:237] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:06.042524 26984 raft_consensus.cc:1261] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:10:06.043957 26036 raft_consensus.cc:1261] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 11:10:06.044860 27556 consensus_queue.cc:1035] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:10:06.049208 27705 consensus_queue.cc:1035] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:10:06.051285 27705 raft_consensus.cc:2932] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:06.052750 26986 raft_consensus.cc:2932] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:06.059028 26036 raft_consensus.cc:2932] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:06.066000 25637 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 3c69ebea3aeb47d48ab8e083fdf71f7c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 11:10:06.070370 25658 catalog_manager.cc:5495] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: config changed from index 3 to 4, VOTER 606d49e053a346e486f48cb31832b815 (127.23.120.129) evicted. New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:06.083428 25785 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c69ebea3aeb47d48ab8e083fdf71f7c with delete_type TABLET_DATA_TOMBSTONED (TS 606d49e053a346e486f48cb31832b815 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53306
I20260807 11:10:06.084489 27685 tablet_replica.cc:330] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 606d49e053a346e486f48cb31832b815: stopping tablet replica
I20260807 11:10:06.085541 27685 raft_consensus.cc:2229] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:10:06.086267 27685 raft_consensus.cc:2258] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:10:06.090834 27685 ts_tablet_manager.cc:1905] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 606d49e053a346e486f48cb31832b815: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 11:10:06.107685 27685 ts_tablet_manager.cc:1918] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 606d49e053a346e486f48cb31832b815: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 11:10:06.108059 27685 log.cc:1196] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 606d49e053a346e486f48cb31832b815: Deleting WAL directory at /tmp/dist-test-task2JM9WS/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786100902301536-24034-0/minicluster-data/ts-0-root/wals/3c69ebea3aeb47d48ab8e083fdf71f7c
I20260807 11:10:06.109383 25637 catalog_manager.cc:4841] TS 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): tablet 3c69ebea3aeb47d48ab8e083fdf71f7c (table test-workload [id=973419b8f3794d3a90f0b3985e9c0d7e]) successfully deleted
I20260807 11:10:06.726451 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260807 11:10:07.058645 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 151266us
W20260807 11:10:07.159364 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 98084us
W20260807 11:10:07.365880 25872 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.688s user 0.010s sys 0.018s
W20260807 11:10:07.366258 25872 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.688s user 0.010s sys 0.018s
W20260807 11:10:07.366559 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 688853us
I20260807 11:10:07.399829 25920 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c"
dest_uuid: "4fd6381544574ddca559ce63b9243efc"
mode: GRACEFUL
new_leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
from {username='slave'} at 127.0.0.1:34992
I20260807 11:10:07.400617 25920 raft_consensus.cc:595] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Received request to transfer leadership to TS eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:07.428424 25800 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb"
dest_uuid: "606d49e053a346e486f48cb31832b815"
mode: GRACEFUL
new_leader_uuid: "4fd6381544574ddca559ce63b9243efc"
from {username='slave'} at 127.0.0.1:58114
I20260807 11:10:07.429080 25800 raft_consensus.cc:595] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Received request to transfer leadership to TS 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:07.437216 25920 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349"
dest_uuid: "4fd6381544574ddca559ce63b9243efc"
mode: GRACEFUL
new_leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
from {username='slave'} at 127.0.0.1:34992
I20260807 11:10:07.437898 25920 raft_consensus.cc:595] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Received request to transfer leadership to TS eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:07.458418 27726 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:07.459056 27726 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:07.461221 27726 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:07.479883 27750 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:07.480504 27750 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:07.482921 27750 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:07.482969 27734 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:07.483552 27734 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:07.485778 27734 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:07.510982 27734 raft_consensus.cc:979] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc: : Instructing follower eb1fca62dcb34ed9b32188ccb19f0543 to start an election
I20260807 11:10:07.511477 27734 raft_consensus.cc:1067] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Signalling peer eb1fca62dcb34ed9b32188ccb19f0543 to start an election
I20260807 11:10:07.513795 27734 raft_consensus.cc:979] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc: : Instructing follower eb1fca62dcb34ed9b32188ccb19f0543 to start an election
I20260807 11:10:07.514215 27734 raft_consensus.cc:1067] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Signalling peer eb1fca62dcb34ed9b32188ccb19f0543 to start an election
I20260807 11:10:07.517680 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:07.518347 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:07.528558 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:07.530131 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:07.533486 26985 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349"
dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
from {username='slave'} at 127.0.0.1:50942
I20260807 11:10:07.534049 26985 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:10:07.534376 26985 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:07.535050 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:07.546296 26985 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:07.555090 26979 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c"
dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
from {username='slave'} at 127.0.0.1:50942
I20260807 11:10:07.555703 26979 raft_consensus.cc:482] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:10:07.556047 26979 raft_consensus.cc:3037] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:07.556696 25869 leader_election.cc:302] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:07.561614 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:07.564698 27734 raft_consensus.cc:2726] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:07.565882 25869 leader_election.cc:302] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:07.566569 27734 raft_consensus.cc:2726] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:07.569839 26979 raft_consensus.cc:504] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:07.576639 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:07.577262 26042 raft_consensus.cc:3037] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:07.578001 26985 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:07.579470 27740 raft_consensus.cc:979] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815: : Instructing follower 4fd6381544574ddca559ce63b9243efc to start an election
I20260807 11:10:07.579945 27740 raft_consensus.cc:1067] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Signalling peer 4fd6381544574ddca559ce63b9243efc to start an election
I20260807 11:10:07.581991 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:07.582607 26034 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:07.589193 26042 raft_consensus.cc:2454] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:07.590481 26938 leader_election.cc:302] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:07.590849 26034 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:07.599184 25919 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb"
dest_uuid: "4fd6381544574ddca559ce63b9243efc"
from {username='slave'} at 127.0.0.1:39742
I20260807 11:10:07.598958 25918 consensus_queue.cc:237] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:07.599727 25919 raft_consensus.cc:482] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:10:07.600093 25919 raft_consensus.cc:3037] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:07.601754 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:07.601899 25981 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:07.602344 25916 raft_consensus.cc:3032] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:07.602643 25916 raft_consensus.cc:729] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:07.603348 25916 consensus_queue.cc:260] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:07.604161 27755 raft_consensus.cc:2726] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:07.604694 25916 raft_consensus.cc:3037] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:07.606835 26034 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac"
dest_uuid: "b22d69db45194908ad3d345eba9badd9"
mode: GRACEFUL
new_leader_uuid: "4fd6381544574ddca559ce63b9243efc"
from {username='slave'} at 127.0.0.1:58132
I20260807 11:10:07.607566 26034 raft_consensus.cc:595] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Received request to transfer leadership to TS 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:07.609229 25919 raft_consensus.cc:504] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:07.610363 25932 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d5d5c9aff2844213a67d710585224083"
dest_uuid: "4fd6381544574ddca559ce63b9243efc"
mode: GRACEFUL
new_leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
from {username='slave'} at 127.0.0.1:34992
I20260807 11:10:07.598914 26979 leader_election.cc:288] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:07.610881 25932 raft_consensus.cc:595] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Received request to transfer leadership to TS eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:07.611630 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:07.612186 25800 raft_consensus.cc:3032] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:07.612450 25800 raft_consensus.cc:729] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:07.613483 25932 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58"
dest_uuid: "4fd6381544574ddca559ce63b9243efc"
mode: GRACEFUL
new_leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
from {username='slave'} at 127.0.0.1:34992
I20260807 11:10:07.613216 25800 consensus_queue.cc:260] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:07.614023 25932 raft_consensus.cc:595] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Received request to transfer leadership to TS eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:07.614554 25800 raft_consensus.cc:3037] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:07.615336 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260807 11:10:07.617071 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:10:07.617765 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:10:07.621882 25919 leader_election.cc:288] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:07.622850 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:07.623400 26034 raft_consensus.cc:3037] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:07.625471 25916 raft_consensus.cc:2454] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:07.629626 25800 raft_consensus.cc:2454] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:07.630883 25869 leader_election.cc:302] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:07.631631 27734 raft_consensus.cc:2781] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:07.632097 27734 raft_consensus.cc:686] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:07.632926 27734 consensus_queue.cc:237] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:07.634790 27757 raft_consensus.cc:2781] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:07.635674 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:10:07.635504 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 45110.4us
I20260807 11:10:07.636293 25916 raft_consensus.cc:3032] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:07.636643 25916 raft_consensus.cc:729] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:07.637132 26938 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:07.637396 25916 consensus_queue.cc:260] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:07.641265 25916 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
W20260807 11:10:07.641782 25869 consensus_peers.cc:487] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f40bb008c149418989fdcd7c1e67d554. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:10:07.644425 26037 raft_consensus.cc:1261] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:10:07.646068 27754 consensus_queue.cc:1035] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:10:07.659785 25800 raft_consensus.cc:1261] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:10:07.661435 27754 consensus_queue.cc:1035] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:10:07.665846 27757 raft_consensus.cc:686] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:07.676110 25916 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:07.682117 27763 raft_consensus.cc:2781] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:07.681186 26034 raft_consensus.cc:2454] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:07.688743 27763 raft_consensus.cc:686] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:07.690155 27763 consensus_queue.cc:237] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:07.673563 27757 consensus_queue.cc:237] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:07.703410 27754 raft_consensus.cc:2932] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:07.711349 25800 raft_consensus.cc:2932] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:07.734015 25658 catalog_manager.cc:5495] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: term changed from 1 to 2, leader changed from 4fd6381544574ddca559ce63b9243efc (127.23.120.130) to eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132). New cstate: current_term: 2 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:10:07.734797 26037 raft_consensus.cc:2932] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:07.736410 25796 consensus_queue.cc:237] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:07.739904 25654 catalog_manager.cc:5495] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 1 to 2, leader changed from 606d49e053a346e486f48cb31832b815 (127.23.120.129) to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 2 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
W20260807 11:10:07.758451 25750 consensus_peers.cc:487] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 66d10b6b12404f0488b9f83c8080069f. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:10:07.762173 25916 raft_consensus.cc:1261] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:10:07.762527 25660 catalog_manager.cc:5495] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:07.766244 26037 raft_consensus.cc:1261] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:10:07.766654 27739 consensus_queue.cc:1035] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:07.776863 27739 consensus_queue.cc:1035] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:07.807538 27750 raft_consensus.cc:979] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc: : Instructing follower eb1fca62dcb34ed9b32188ccb19f0543 to start an election
I20260807 11:10:07.808076 27750 raft_consensus.cc:1067] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Signalling peer eb1fca62dcb34ed9b32188ccb19f0543 to start an election
I20260807 11:10:07.812690 26979 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d5d5c9aff2844213a67d710585224083"
dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
from {username='slave'} at 127.0.0.1:50942
I20260807 11:10:07.813383 26979 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:10:07.813771 26979 raft_consensus.cc:3037] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:07.823619 25916 raft_consensus.cc:2932] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:07.820779 27739 raft_consensus.cc:2932] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:07.832341 26036 raft_consensus.cc:2932] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:07.838189 25654 catalog_manager.cc:5495] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: term changed from 1 to 2, leader changed from 4fd6381544574ddca559ce63b9243efc (127.23.120.130) to eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132). New cstate: current_term: 2 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:10:07.857918 25916 consensus_queue.cc:237] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:07.867543 26979 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:07.871044 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:07.871651 25918 raft_consensus.cc:3032] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:07.871937 25918 raft_consensus.cc:729] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:07.868669 25658 catalog_manager.cc:5495] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:07.872610 25918 consensus_queue.cc:260] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:07.873721 25918 raft_consensus.cc:3037] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:07.878191 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:07.878824 26037 raft_consensus.cc:3037] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:07.887817 27733 raft_consensus.cc:979] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc: : Instructing follower eb1fca62dcb34ed9b32188ccb19f0543 to start an election
I20260807 11:10:07.888366 27733 raft_consensus.cc:1067] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Signalling peer eb1fca62dcb34ed9b32188ccb19f0543 to start an election
I20260807 11:10:07.890939 26987 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58"
dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
from {username='slave'} at 127.0.0.1:50942
I20260807 11:10:07.891577 26987 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:10:07.891968 26987 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:07.900017 26037 raft_consensus.cc:2454] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:07.910005 26979 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:07.910460 26938 leader_election.cc:302] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:07.911116 25918 raft_consensus.cc:2454] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:07.911397 27757 raft_consensus.cc:2781] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:07.912050 27757 raft_consensus.cc:686] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:07.912986 27757 consensus_queue.cc:237] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:07.916034 25869 consensus_peers.cc:487] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7be451f12275472a9c2c53877d9d1a0a. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:10:07.916350 26037 raft_consensus.cc:1261] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:10:07.917812 27734 consensus_queue.cc:1035] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:10:07.908100 26987 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:07.926129 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:07.926790 25796 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:07.934334 25796 raft_consensus.cc:2454] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:07.937358 26936 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:07.938215 27763 raft_consensus.cc:2781] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:07.938848 27763 raft_consensus.cc:686] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
W20260807 11:10:07.944257 25702 debug-util.cc:398] Leaking SignalData structure 0x7b0800094700 after lost signal to thread 24038
I20260807 11:10:07.944259 25800 raft_consensus.cc:1261] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 11:10:07.945641 25702 debug-util.cc:398] Leaking SignalData structure 0x7b0800033c00 after lost signal to thread 25705
W20260807 11:10:07.947160 25702 debug-util.cc:398] Leaking SignalData structure 0x7b08002ba240 after lost signal to thread 25850
I20260807 11:10:07.947696 27733 consensus_queue.cc:1035] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260807 11:10:07.948717 25702 debug-util.cc:398] Leaking SignalData structure 0x7b0800334aa0 after lost signal to thread 25964
I20260807 11:10:07.956063 25658 catalog_manager.cc:5495] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: term changed from 1 to 2, leader changed from 4fd6381544574ddca559ce63b9243efc (127.23.120.130) to eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132). New cstate: current_term: 2 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:10:07.940009 27763 consensus_queue.cc:237] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:07.956933 26987 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:10:07.965081 25702 debug-util.cc:398] Leaking SignalData structure 0x7b08003169e0 after lost signal to thread 26083
W20260807 11:10:07.970256 25702 debug-util.cc:398] Leaking SignalData structure 0x7b08001572c0 after lost signal to thread 27031
I20260807 11:10:07.974021 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:07.985602 25916 raft_consensus.cc:3032] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:07.985984 25916 raft_consensus.cc:729] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:07.991840 25916 consensus_queue.cc:260] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:07.996640 25916 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:08.006978 25916 raft_consensus.cc:2454] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:08.019773 25658 catalog_manager.cc:5495] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: term changed from 1 to 2, leader changed from 4fd6381544574ddca559ce63b9243efc (127.23.120.130) to eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132). New cstate: current_term: 2 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:10:08.020889 27733 raft_consensus.cc:2932] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:08.032477 25800 raft_consensus.cc:2932] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:08.031205 26037 raft_consensus.cc:2932] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:08.050294 27793 ts_tablet_manager.cc:927] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:08.055069 25796 consensus_queue.cc:237] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:08.059962 25658 catalog_manager.cc:5495] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 11:10:08.077608 26036 raft_consensus.cc:1261] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:10:08.078395 27743 raft_consensus.cc:979] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9: : Instructing follower 4fd6381544574ddca559ce63b9243efc to start an election
W20260807 11:10:08.080120 25750 consensus_peers.cc:487] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 -> Peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279): Couldn't send request to peer eb1fca62dcb34ed9b32188ccb19f0543. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf86a21774c34f33a6540d1dad0de0c4. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:10:08.083665 27724 raft_consensus.cc:1067] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Signalling peer 4fd6381544574ddca559ce63b9243efc to start an election
I20260807 11:10:08.084050 25918 raft_consensus.cc:1261] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:10:08.082093 27736 consensus_queue.cc:1035] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260807 11:10:08.107537 25918 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac"
dest_uuid: "4fd6381544574ddca559ce63b9243efc"
from {username='slave'} at 127.0.0.1:39752
I20260807 11:10:08.108295 25918 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:10:08.108644 25918 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:08.106374 27739 consensus_queue.cc:1035] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260807 11:10:08.113824 25796 raft_consensus.cc:1261] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:10:08.118822 25918 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:08.121450 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:08.120973 25918 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:08.122081 26036 raft_consensus.cc:3032] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:08.122447 26036 raft_consensus.cc:729] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:08.122272 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:08.123950 25796 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:08.123665 26036 consensus_queue.cc:260] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:08.124732 26036 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:08.120806 27712 consensus_queue.cc:1035] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260807 11:10:08.132579 26036 raft_consensus.cc:2454] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:08.133891 25872 leader_election.cc:302] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:08.134824 27712 raft_consensus.cc:2781] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:08.135288 27712 raft_consensus.cc:686] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:08.136215 27712 consensus_queue.cc:237] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:08.132237 27793 tablet_copy_client.cc:316] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.130:36465
I20260807 11:10:08.138413 25796 raft_consensus.cc:2454] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:08.140146 25945 tablet_copy_service.cc:140] P 4fd6381544574ddca559ce63b9243efc: Received BeginTabletCopySession request for tablet f40bb008c149418989fdcd7c1e67d554 from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:35276)
I20260807 11:10:08.140739 25945 tablet_copy_service.cc:161] P 4fd6381544574ddca559ce63b9243efc: Beginning new tablet copy session on tablet f40bb008c149418989fdcd7c1e67d554 from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:35276: session id = eb1fca62dcb34ed9b32188ccb19f0543-f40bb008c149418989fdcd7c1e67d554
I20260807 11:10:08.158092 25916 raft_consensus.cc:1261] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:08.161067 27763 consensus_queue.cc:1035] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 11:10:08.169974 25658 catalog_manager.cc:5495] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 1 to 2, leader changed from b22d69db45194908ad3d345eba9badd9 (127.23.120.131) to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 2 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:10:08.175257 25945 tablet_copy_source_session.cc:205] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc: Tablet Copy: opened 0 blocks and 1 log segments
W20260807 11:10:08.175508 25960 debug-util.cc:398] Leaking SignalData structure 0x7b08001698a0 after lost signal to thread 24038
W20260807 11:10:08.195560 25960 debug-util.cc:398] Leaking SignalData structure 0x7b0800169f80 after lost signal to thread 25705
I20260807 11:10:08.196727 26034 raft_consensus.cc:1261] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260807 11:10:08.197342 25960 debug-util.cc:398] Leaking SignalData structure 0x7b0800329500 after lost signal to thread 25850
I20260807 11:10:08.198436 27734 consensus_queue.cc:1035] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260807 11:10:08.199071 25960 debug-util.cc:398] Leaking SignalData structure 0x7b08002ad180 after lost signal to thread 25964
W20260807 11:10:08.200681 25960 debug-util.cc:398] Leaking SignalData structure 0x7b080031cf20 after lost signal to thread 26083
I20260807 11:10:08.211920 27793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f40bb008c149418989fdcd7c1e67d554. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:10:08.235940 25932 raft_consensus.cc:1261] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:08.236411 26036 raft_consensus.cc:2932] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:08.235785 27741 raft_consensus.cc:2932] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
I20260807 11:10:08.243937 27763 consensus_queue.cc:1035] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 11:10:08.261929 26037 raft_consensus.cc:1261] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:08.265071 26042 raft_consensus.cc:1261] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:08.267096 27763 consensus_queue.cc:1035] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260807 11:10:08.258260 25960 debug-util.cc:398] Leaking SignalData structure 0x7b08002902c0 after lost signal to thread 27031
I20260807 11:10:08.286526 27762 consensus_queue.cc:1035] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260807 11:10:08.296537 26037 raft_consensus.cc:1261] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:08.298632 27757 consensus_queue.cc:1035] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
W20260807 11:10:08.321056 27805 log.cc:924] Time spent T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
I20260807 11:10:08.329674 25916 raft_consensus.cc:1261] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:08.336505 27763 consensus_queue.cc:1035] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 11:10:08.346055 25918 raft_consensus.cc:2932] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } } }
W20260807 11:10:08.434942 27809 log.cc:924] Time spent T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.074s user 0.000s sys 0.001s
I20260807 11:10:08.474743 27819 ts_tablet_manager.cc:927] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:08.522598 25796 raft_consensus.cc:1261] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:08.534217 25920 raft_consensus.cc:1261] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:08.553270 25800 raft_consensus.cc:1261] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:10:08.552471 25658 catalog_manager.cc:5495] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 reported cstate change: config changed from index -1 to 2, NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) added. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 11:10:08.562096 27520 consensus_queue.cc:1035] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:10:08.601457 27821 ts_tablet_manager.cc:927] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195)
I20260807 11:10:08.532642 27763 consensus_queue.cc:1035] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
W20260807 11:10:08.617656 27816 log.cc:924] Time spent T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20260807 11:10:08.672991 27818 consensus_queue.cc:1035] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260807 11:10:08.664485 27793 tablet_copy_client.cc:791] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:10:08.709722 27793 tablet_copy_client.cc:655] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
W20260807 11:10:08.782491 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 270795us
W20260807 11:10:08.808908 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 275272us
I20260807 11:10:08.809450 26037 raft_consensus.cc:1261] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:10:08.834561 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260807 11:10:08.845165 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 204871us
W20260807 11:10:08.846278 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 61146us
I20260807 11:10:08.879242 27821 tablet_copy_client.cc:316] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.129:40195
I20260807 11:10:08.916908 27819 tablet_copy_client.cc:316] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.130:36465
W20260807 11:10:09.071276 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 155438us
W20260807 11:10:09.097617 25755 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.045s sys 0.100s
I20260807 11:10:09.090157 27793 tablet_copy_client.cc:531] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260807 11:10:09.100242 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 238478us
I20260807 11:10:09.081286 25944 tablet_copy_service.cc:140] P 4fd6381544574ddca559ce63b9243efc: Received BeginTabletCopySession request for tablet 7be451f12275472a9c2c53877d9d1a0a from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:35276)
W20260807 11:10:09.103354 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 237708us
I20260807 11:10:09.134763 27520 consensus_queue.cc:1035] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260807 11:10:09.141234 25755 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.585s user 0.045s sys 0.101s
W20260807 11:10:09.128700 26938 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.091s sys 0.115s
I20260807 11:10:09.133317 25944 tablet_copy_service.cc:161] P 4fd6381544574ddca559ce63b9243efc: Beginning new tablet copy session on tablet 7be451f12275472a9c2c53877d9d1a0a from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:35276: session id = eb1fca62dcb34ed9b32188ccb19f0543-7be451f12275472a9c2c53877d9d1a0a
W20260807 11:10:09.160036 26938 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.549s user 0.092s sys 0.115s
W20260807 11:10:09.172640 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 569329us
W20260807 11:10:09.179944 27836 log.cc:924] Time spent T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.113s user 0.000s sys 0.001s
I20260807 11:10:09.204543 25944 tablet_copy_source_session.cc:205] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:10:09.225246 27853 ts_tablet_manager.cc:927] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195)
W20260807 11:10:09.272361 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 59413.2us
I20260807 11:10:09.280382 26034 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65"
dest_uuid: "b22d69db45194908ad3d345eba9badd9"
mode: GRACEFUL
new_leader_uuid: "606d49e053a346e486f48cb31832b815"
from {username='slave'} at 127.0.0.1:58132
I20260807 11:10:09.281083 26034 raft_consensus.cc:595] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Received request to transfer leadership to TS 606d49e053a346e486f48cb31832b815
W20260807 11:10:09.286559 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 194716us
I20260807 11:10:09.299683 27793 tablet_bootstrap.cc:492] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
W20260807 11:10:09.322995 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 189045us
I20260807 11:10:09.393021 27819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be451f12275472a9c2c53877d9d1a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:10:09.399753 25825 tablet_copy_service.cc:140] P 606d49e053a346e486f48cb31832b815: Received BeginTabletCopySession request for tablet 66d10b6b12404f0488b9f83c8080069f from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:59118)
I20260807 11:10:09.407775 25825 tablet_copy_service.cc:161] P 606d49e053a346e486f48cb31832b815: Beginning new tablet copy session on tablet 66d10b6b12404f0488b9f83c8080069f from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:59118: session id = eb1fca62dcb34ed9b32188ccb19f0543-66d10b6b12404f0488b9f83c8080069f
W20260807 11:10:09.452594 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 176259us
I20260807 11:10:09.456032 25800 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2"
dest_uuid: "606d49e053a346e486f48cb31832b815"
mode: GRACEFUL
new_leader_uuid: "4fd6381544574ddca559ce63b9243efc"
from {username='slave'} at 127.0.0.1:58114
I20260807 11:10:09.456660 25800 raft_consensus.cc:595] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Received request to transfer leadership to TS 4fd6381544574ddca559ce63b9243efc
W20260807 11:10:09.457480 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 80399.9us
W20260807 11:10:09.459311 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 74391.3us
W20260807 11:10:09.475339 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 91057.3us
I20260807 11:10:09.460038 27853 tablet_copy_client.cc:316] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Beginning tablet copy session from remote peer at address 127.23.120.129:40195
I20260807 11:10:09.495705 25799 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d8012276091e4b63b062cb7152208a16"
dest_uuid: "606d49e053a346e486f48cb31832b815"
mode: GRACEFUL
new_leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
from {username='slave'} at 127.0.0.1:58114
I20260807 11:10:09.501724 25799 raft_consensus.cc:595] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Received request to transfer leadership to TS eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:09.510102 27858 log.cc:924] Time spent T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.172s user 0.000s sys 0.001s
W20260807 11:10:09.522470 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 59465.8us
I20260807 11:10:09.530684 25801 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7"
dest_uuid: "606d49e053a346e486f48cb31832b815"
mode: GRACEFUL
new_leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
from {username='slave'} at 127.0.0.1:58114
I20260807 11:10:09.531276 25801 raft_consensus.cc:595] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Received request to transfer leadership to TS eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:09.547717 27859 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:09.548426 25822 tablet_copy_service.cc:140] P 606d49e053a346e486f48cb31832b815: Received BeginTabletCopySession request for tablet cf86a21774c34f33a6540d1dad0de0c4 from peer eb1fca62dcb34ed9b32188ccb19f0543 ({username='slave'} at 127.0.0.1:59118)
I20260807 11:10:09.548269 27859 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:09.549024 25822 tablet_copy_service.cc:161] P 606d49e053a346e486f48cb31832b815: Beginning new tablet copy session on tablet cf86a21774c34f33a6540d1dad0de0c4 from peer eb1fca62dcb34ed9b32188ccb19f0543 at {username='slave'} at 127.0.0.1:59118: session id = eb1fca62dcb34ed9b32188ccb19f0543-cf86a21774c34f33a6540d1dad0de0c4
I20260807 11:10:09.561041 25932 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf"
dest_uuid: "4fd6381544574ddca559ce63b9243efc"
mode: GRACEFUL
new_leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
from {username='slave'} at 127.0.0.1:34992
I20260807 11:10:09.561642 25932 raft_consensus.cc:595] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Received request to transfer leadership to TS eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:09.564231 27859 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:09.572346 27862 raft_consensus.cc:482] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:09.579764 27733 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:09.580368 27733 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:09.580559 27862 raft_consensus.cc:504] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:09.581893 25932 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f75da3017fd744719647332808daba49"
dest_uuid: "4fd6381544574ddca559ce63b9243efc"
mode: GRACEFUL
new_leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
from {username='slave'} at 127.0.0.1:34992
I20260807 11:10:09.582422 25932 raft_consensus.cc:595] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Received request to transfer leadership to TS eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:09.582968 27733 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:09.586587 27863 raft_consensus.cc:979] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc: : Instructing follower eb1fca62dcb34ed9b32188ccb19f0543 to start an election
I20260807 11:10:09.586840 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260807 11:10:09.586643 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:09.593087 27714 raft_consensus.cc:979] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9: : Instructing follower 606d49e053a346e486f48cb31832b815 to start an election
I20260807 11:10:09.593636 27714 raft_consensus.cc:1067] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Signalling peer 606d49e053a346e486f48cb31832b815 to start an election
I20260807 11:10:09.612005 27879 raft_consensus.cc:979] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc: : Instructing follower eb1fca62dcb34ed9b32188ccb19f0543 to start an election
I20260807 11:10:09.612701 27879 raft_consensus.cc:1067] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Signalling peer eb1fca62dcb34ed9b32188ccb19f0543 to start an election
I20260807 11:10:09.619289 27882 raft_consensus.cc:1067] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Signalling peer eb1fca62dcb34ed9b32188ccb19f0543 to start an election
I20260807 11:10:09.621549 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:09.625684 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:09.633006 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:10:09.633690 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:10:09.641124 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:09.646463 25869 leader_election.cc:302] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:09.658494 27733 raft_consensus.cc:2726] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:09.670257 27862 leader_election.cc:288] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:09.676200 27856 raft_consensus.cc:979] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815: : Instructing follower 4fd6381544574ddca559ce63b9243efc to start an election
I20260807 11:10:09.658900 27712 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
W20260807 11:10:09.690544 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 44222.5us
I20260807 11:10:09.700405 27854 raft_consensus.cc:1067] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Signalling peer 4fd6381544574ddca559ce63b9243efc to start an election
I20260807 11:10:09.654837 27882 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:09.697969 25752 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:09.707428 26987 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf"
dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
from {username='slave'} at 127.0.0.1:50942
I20260807 11:10:09.708151 26987 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:10:09.708549 26987 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:09.712666 27882 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:09.718570 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:09.718724 26985 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f75da3017fd744719647332808daba49"
dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
from {username='slave'} at 127.0.0.1:50942
I20260807 11:10:09.720176 26985 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:10:09.720546 26985 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:09.722254 26987 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:09.728293 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:09.733088 25803 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65"
dest_uuid: "606d49e053a346e486f48cb31832b815"
from {username='slave'} at 127.0.0.1:53324
I20260807 11:10:09.733762 25803 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:10:09.736086 25803 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:09.739571 26987 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:09.739854 25750 leader_election.cc:302] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:09.727008 27740 raft_consensus.cc:2726] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:09.752422 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:09.762456 25918 raft_consensus.cc:3032] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:09.762864 25918 raft_consensus.cc:729] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:09.764196 25920 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2"
dest_uuid: "4fd6381544574ddca559ce63b9243efc"
from {username='slave'} at 127.0.0.1:39742
I20260807 11:10:09.765035 25920 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:10:09.765559 25920 raft_consensus.cc:3037] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:09.751634 27856 raft_consensus.cc:2726] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:09.773795 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:09.774560 25918 consensus_queue.cc:260] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:09.774588 25806 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:09.776026 25918 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:09.701690 27712 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:09.779420 25803 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:09.805929 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:09.806622 26036 raft_consensus.cc:3032] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:09.806974 26036 raft_consensus.cc:729] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:09.807933 26036 consensus_queue.cc:260] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:09.819288 26036 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:09.829353 25803 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:09.830336 25806 raft_consensus.cc:2454] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:09.841600 26985 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:09.843921 25920 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:09.846714 26987 raft_consensus.cc:1226] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:09.851033 25920 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:09.855214 26936 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:09.856679 26985 leader_election.cc:288] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:09.858170 25918 raft_consensus.cc:2454] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:09.873601 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:09.877260 25803 raft_consensus.cc:3032] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:09.877585 25803 raft_consensus.cc:729] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:09.882800 25803 consensus_queue.cc:260] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:09.883074 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:09.889694 26036 raft_consensus.cc:2454] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:09.903405 25803 raft_consensus.cc:3037] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:09.907963 25800 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:09.915066 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:09.915839 25932 raft_consensus.cc:3032] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:09.916239 25932 raft_consensus.cc:729] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:09.924574 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:09.925390 26985 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:09.934142 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:09.935398 26980 raft_consensus.cc:3037] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:09.884654 27839 raft_consensus.cc:2781] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:09.957837 25803 raft_consensus.cc:2454] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:09.961585 25752 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:09.966589 25869 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:09.923947 25932 consensus_queue.cc:260] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:09.976970 25932 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:09.982470 27712 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:09.970886 27839 raft_consensus.cc:686] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:09.990299 27839 consensus_queue.cc:237] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:09.990314 27862 raft_consensus.cc:2781] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:10.009776 25800 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
W20260807 11:10:10.012285 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 45826.5us
I20260807 11:10:10.020788 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:10.021270 26985 raft_consensus.cc:2454] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:09.975739 25825 tablet_copy_source_session.cc:205] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:10:10.017549 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:10.025775 25932 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:10.026279 27712 raft_consensus.cc:2781] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:10.034303 26936 leader_election.cc:302] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:10.023123 25822 tablet_copy_source_session.cc:205] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:10:10.036949 27882 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:10.062889 27842 raft_consensus.cc:2781] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:10.052060 27712 raft_consensus.cc:686] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
W20260807 11:10:10.070485 27865 log.cc:924] Time spent T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20260807 11:10:10.074359 27739 raft_consensus.cc:979] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815: : Instructing follower eb1fca62dcb34ed9b32188ccb19f0543 to start an election
I20260807 11:10:10.075006 27739 raft_consensus.cc:1067] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Signalling peer eb1fca62dcb34ed9b32188ccb19f0543 to start an election
I20260807 11:10:10.077157 26979 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d8012276091e4b63b062cb7152208a16"
dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
from {username='slave'} at 127.0.0.1:50958
I20260807 11:10:10.077761 26979 raft_consensus.cc:482] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:10:10.078162 26979 raft_consensus.cc:3037] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:10.079394 26980 raft_consensus.cc:2454] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
W20260807 11:10:10.084942 27892 log.cc:924] Time spent T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.134s user 0.000s sys 0.001s
I20260807 11:10:10.086227 26979 raft_consensus.cc:504] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:10.064236 27842 raft_consensus.cc:686] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:10.106388 27853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf86a21774c34f33a6540d1dad0de0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:10:10.117728 27712 consensus_queue.cc:237] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:10.121930 27842 consensus_queue.cc:237] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:10.151925 27819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:50942 (request call id 241) took 1830 ms. Trace:
I20260807 11:10:10.152561 27819 rpcz_store.cc:276] 0807 11:10:08.321143 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:08.321328 (+ 185us) service_pool.cc:227] Handling call
0807 11:10:08.475268 (+153940us) ts_tablet_manager.cc:928] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
0807 11:10:09.259176 (+783908us) tablet_copy_client.cc:361] Tablet copy session begun
0807 11:10:09.757548 (+498372us) tablet_metadata.cc:641] Metadata flushed
0807 11:10:09.758069 (+ 521us) tablet_copy_client.cc:490] Wrote new tablet metadata
0807 11:10:10.140356 (+382287us) tablet_copy_client.cc:411] Wrote new consensus metadata
0807 11:10:10.151244 (+ 10888us) tablet_replica.cc:205] Creating consensus instance
0807 11:10:10.151694 (+ 450us) ts_tablet_manager.cc:983] Replica registered.
0807 11:10:10.151779 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":59449,"tablet-copy.queue_time_us":101710,"thread_start_us":440732,"threads_started":2}
I20260807 11:10:10.203399 27819 tablet_copy_client.cc:791] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:10:10.224562 27793 tablet_bootstrap.cc:492] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:10:10.255236 27819 tablet_copy_client.cc:655] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:10:10.257385 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:10.258008 25800 raft_consensus.cc:3032] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:10.258306 25800 raft_consensus.cc:729] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:10.258977 25800 consensus_queue.cc:260] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:10.260038 25800 raft_consensus.cc:3037] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:10.266237 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:10.293130 26979 leader_election.cc:288] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:10.261129 27793 tablet_bootstrap.cc:492] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:10:10.298542 27793 ts_tablet_manager.cc:1397] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 1.010s user 0.054s sys 0.049s
I20260807 11:10:10.348366 25800 raft_consensus.cc:2454] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:10.350070 26936 leader_election.cc:302] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:10.370117 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:10.335970 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 429086us
I20260807 11:10:10.405743 27821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66d10b6b12404f0488b9f83c8080069f. 1 dirs total, 0 dirs full, 0 dirs failed
W20260807 11:10:10.410778 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 381536us
I20260807 11:10:10.422664 25869 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:10.429448 27817 raft_consensus.cc:2726] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:10.434223 25869 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:10.431106 27862 raft_consensus.cc:686] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 2 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:10.443516 27819 tablet_copy_client.cc:531] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260807 11:10:10.465338 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 460219us
I20260807 11:10:10.456920 27882 raft_consensus.cc:2726] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:10.515260 27862 consensus_queue.cc:237] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:10.547704 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:10.492296 27843 raft_consensus.cc:2781] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:10.563027 26038 raft_consensus.cc:3037] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
W20260807 11:10:10.539045 25981 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.016s sys 0.070s
W20260807 11:10:10.599077 25981 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.647s user 0.018s sys 0.071s
W20260807 11:10:10.599493 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 659649us
I20260807 11:10:10.600554 27843 raft_consensus.cc:686] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:10.549950 27875 raft_consensus.cc:482] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:10.567951 27834 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:10.639892 25786 raft_consensus.cc:1261] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:10.656102 26038 raft_consensus.cc:2454] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:10.601611 27843 consensus_queue.cc:237] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:10.672852 27918 log.cc:924] Time spent T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20260807 11:10:10.658118 27762 consensus_queue.cc:1035] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260807 11:10:10.649678 27834 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:10.710911 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 98670.6us
I20260807 11:10:10.617651 27875 raft_consensus.cc:504] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:10.726785 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 59857.7us
W20260807 11:10:10.728519 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 60293.9us
I20260807 11:10:10.753835 26985 raft_consensus.cc:1261] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:10.767570 27851 consensus_queue.cc:1035] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 11:10:10.770514 27834 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:10:10.779661 27913 log.cc:924] Time spent T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.174s user 0.000s sys 0.001s
I20260807 11:10:10.742354 27890 raft_consensus.cc:979] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815: : Instructing follower eb1fca62dcb34ed9b32188ccb19f0543 to start an election
W20260807 11:10:10.826376 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 76651.2us
I20260807 11:10:10.785472 27819 tablet_bootstrap.cc:492] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
W20260807 11:10:10.838793 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 88593.3us
I20260807 11:10:10.858357 25796 raft_consensus.cc:1261] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:10.860726 27875 leader_election.cc:288] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:10.836522 27741 raft_consensus.cc:1067] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Signalling peer eb1fca62dcb34ed9b32188ccb19f0543 to start an election
I20260807 11:10:10.866318 25796 raft_consensus.cc:1261] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:10.789973 27793 raft_consensus.cc:348] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:10.868968 27733 consensus_queue.cc:1035] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 11:10:10.867883 27793 raft_consensus.cc:729] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:10:10.881855 27907 consensus_queue.cc:1035] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260807 11:10:10.890926 26979 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7"
dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
from {username='slave'} at 127.0.0.1:50958
I20260807 11:10:10.891724 26979 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:10:10.892184 26979 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:10.894507 27853 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:50958 (request call id 255) took 2025 ms. Trace:
I20260807 11:10:10.894948 27853 rpcz_store.cc:276] 0807 11:10:08.889960 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:08.914343 (+ 24383us) service_pool.cc:227] Handling call
0807 11:10:09.225718 (+311375us) ts_tablet_manager.cc:928] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195)
0807 11:10:10.095029 (+869311us) tablet_copy_client.cc:361] Tablet copy session begun
0807 11:10:10.634371 (+539342us) tablet_metadata.cc:641] Metadata flushed
0807 11:10:10.634681 (+ 310us) tablet_copy_client.cc:490] Wrote new tablet metadata
0807 11:10:10.838288 (+203607us) tablet_copy_client.cc:411] Wrote new consensus metadata
0807 11:10:10.884078 (+ 45790us) tablet_replica.cc:205] Creating consensus instance
0807 11:10:10.894351 (+ 10273us) ts_tablet_manager.cc:983] Replica registered.
0807 11:10:10.894435 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":77364,"tablet-copy.queue_time_us":306530,"thread_start_us":362731,"threads_started":2}
I20260807 11:10:10.896214 27853 tablet_copy_client.cc:791] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:10:10.896827 27853 tablet_copy_client.cc:655] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:10:10.858991 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:10:10.921955 25655 catalog_manager.cc:5495] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 1 to 2, leader changed from 606d49e053a346e486f48cb31832b815 (127.23.120.129) to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 2 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 11:10:10.910347 27793 consensus_queue.cc:260] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:10.944587 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:10.984640 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:10.990666 25920 raft_consensus.cc:1261] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:10.987399 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:11.001350 27907 consensus_queue.cc:1035] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260807 11:10:11.046857 26036 raft_consensus.cc:1261] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:11.052350 25981 leader_election.cc:302] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:11.053445 27853 tablet_copy_client.cc:531] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:10:11.057053 27748 raft_consensus.cc:2726] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:11.084846 25801 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "61cb005020884d87b17995a28b8effd6"
dest_uuid: "606d49e053a346e486f48cb31832b815"
mode: GRACEFUL
new_leader_uuid: "4fd6381544574ddca559ce63b9243efc"
from {username='slave'} at 127.0.0.1:58114
I20260807 11:10:11.085697 25801 raft_consensus.cc:595] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Received request to transfer leadership to TS 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:11.096746 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:11.103468 25981 leader_election.cc:302] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:11.076231 27854 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:11.105818 27793 ts_tablet_manager.cc:1428] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.807s user 0.013s sys 0.005s
W20260807 11:10:11.106164 27793 ts_tablet_manager.cc:1460] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543: Tablet startup took 807ms
I20260807 11:10:11.148804 26036 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353"
dest_uuid: "b22d69db45194908ad3d345eba9badd9"
mode: GRACEFUL
new_leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
from {username='slave'} at 127.0.0.1:58132
I20260807 11:10:11.149550 26036 raft_consensus.cc:595] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Received request to transfer leadership to TS eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:11.117264 27793 ts_tablet_manager.cc:1462] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543: Trace:
0807 11:10:08.037131 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:08.037292 (+ 161us) service_pool.cc:227] Handling call
0807 11:10:08.057026 (+ 19734us) ts_tablet_manager.cc:928] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
0807 11:10:08.204601 (+147575us) tablet_copy_client.cc:361] Tablet copy session begun
0807 11:10:08.378970 (+174369us) tablet_metadata.cc:641] Metadata flushed
0807 11:10:08.396042 (+ 17072us) tablet_copy_client.cc:490] Wrote new tablet metadata
0807 11:10:08.632231 (+236189us) tablet_copy_client.cc:411] Wrote new consensus metadata
0807 11:10:08.651211 (+ 18980us) tablet_replica.cc:205] Creating consensus instance
0807 11:10:08.653506 (+ 2295us) ts_tablet_manager.cc:983] Replica registered.
0807 11:10:08.653639 (+ 133us) inbound_call.cc:173] Queueing success response
0807 11:10:09.286224 (+632585us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0807 11:10:10.302500 (+1016276us) ts_tablet_manager.cc:1429] Starting tablet replica
0807 11:10:10.455189 (+152689us) tablet_replica.cc:259] Starting instrumentation
0807 11:10:10.585754 (+130565us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":20627,"mutex_wait_us":32696,"raft.queue_time_us":4085,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":22559,"tablet-copy.queue_time_us":12511,"tablet-download-pool.queue_time_us":11028,"tablet-download-pool.run_cpu_time_us":2178,"tablet-download-pool.run_wall_time_us":360567,"thread_start_us":56897,"threads_started":2}
I20260807 11:10:11.120528 27903 raft_consensus.cc:2726] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:11.059679 27911 consensus_queue.cc:1035] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
I20260807 11:10:11.165572 27821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:50958 (request call id 252) took 2833 ms. Trace:
I20260807 11:10:11.167114 25799 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0"
dest_uuid: "606d49e053a346e486f48cb31832b815"
mode: GRACEFUL
new_leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
from {username='slave'} at 127.0.0.1:58114
I20260807 11:10:11.167802 25799 raft_consensus.cc:595] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Received request to transfer leadership to TS eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:11.105415 27854 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:11.190122 26979 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:11.166085 27821 rpcz_store.cc:276] 0807 11:10:08.332167 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:08.332457 (+ 290us) service_pool.cc:227] Handling call
0807 11:10:08.602037 (+269580us) ts_tablet_manager.cc:928] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195)
0807 11:10:10.104374 (+1502337us) tablet_copy_client.cc:361] Tablet copy session begun
0807 11:10:10.846471 (+742097us) tablet_metadata.cc:641] Metadata flushed
0807 11:10:10.846705 (+ 234us) tablet_copy_client.cc:490] Wrote new tablet metadata
0807 11:10:11.104551 (+257846us) tablet_copy_client.cc:411] Wrote new consensus metadata
0807 11:10:11.164856 (+ 60305us) tablet_replica.cc:205] Creating consensus instance
0807 11:10:11.165346 (+ 490us) ts_tablet_manager.cc:983] Replica registered.
0807 11:10:11.165448 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":492,"tablet-copy.queue_time_us":264179,"thread_start_us":350679,"threads_started":2}
I20260807 11:10:11.806303 27821 tablet_copy_client.cc:791] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 0 data blocks...
I20260807 11:10:11.806921 27821 tablet_copy_client.cc:655] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Starting download of 1 WAL segments...
I20260807 11:10:11.393783 27819 tablet_bootstrap.cc:700] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543: Time spent opening tablet: real 0.362s user 0.000s sys 0.003s
I20260807 11:10:11.816316 26980 raft_consensus.cc:1226] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:11.253603 27854 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:11.886613 26979 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:11.887319 26979 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:50958 (request call id 279) took 1002 ms. Trace:
I20260807 11:10:11.849953 27907 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:11.887689 26979 rpcz_store.cc:276] 0807 11:10:10.884453 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:10.885204 (+ 751us) service_pool.cc:227] Handling call
0807 11:10:11.887225 (+1002021us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:10:11.915876 27954 log.cc:924] Time spent T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260807 11:10:11.918423 27853 tablet_bootstrap.cc:492] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
I20260807 11:10:11.909474 27907 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:11.927156 27907 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.313899 27839 raft_consensus.cc:482] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:12.314495 27839 raft_consensus.cc:504] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:12.316938 27906 raft_consensus.cc:482] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:12.317688 27906 raft_consensus.cc:504] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:12.320279 27906 leader_election.cc:288] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.329912 27839 leader_election.cc:288] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.330595 27839 raft_consensus.cc:482] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:12.331194 27839 raft_consensus.cc:504] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:12.384943 27907 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:12.385715 27907 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:12.388269 27907 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:12.389590 27906 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:12.390262 27906 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:12.396039 27839 leader_election.cc:288] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.396713 27839 raft_consensus.cc:482] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:12.397311 27839 raft_consensus.cc:504] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:12.399684 27839 leader_election.cc:288] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.392685 27906 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.440982 27906 raft_consensus.cc:482] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:12.441776 27906 raft_consensus.cc:504] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:12.444993 27906 leader_election.cc:288] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.633088 27955 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:12.633984 27955 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:12.675207 27907 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:12.675891 27907 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:12.678483 27907 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195)
I20260807 11:10:12.678988 27907 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:12.679478 27907 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:12.681620 27839 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:12.682237 27839 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:12.684271 27906 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:12.684937 27906 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:12.687716 27907 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:12.688274 27907 raft_consensus.cc:482] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:12.688745 27907 raft_consensus.cc:504] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:12.690722 27907 leader_election.cc:288] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.692014 27907 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:12.692528 27907 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:12.692984 27839 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:12.694828 27907 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.694931 27906 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.703979 27819 tablet_bootstrap.cc:492] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:10:12.704870 27819 tablet_bootstrap.cc:492] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:10:12.705778 27955 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.713716 27853 tablet_bootstrap.cc:492] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:10:12.714994 27853 tablet_bootstrap.cc:492] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:10:12.705617 27819 ts_tablet_manager.cc:1397] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 1.921s user 0.018s sys 0.025s
I20260807 11:10:12.719301 27819 raft_consensus.cc:348] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:12.720234 27819 raft_consensus.cc:729] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:10:12.721024 27853 ts_tablet_manager.cc:1397] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 0.803s user 0.015s sys 0.014s
I20260807 11:10:12.724030 27853 raft_consensus.cc:348] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:12.724915 27853 raft_consensus.cc:729] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
I20260807 11:10:12.725600 27853 consensus_queue.cc:260] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:12.726970 27819 consensus_queue.cc:260] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:12.730063 27819 ts_tablet_manager.cc:1428] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.014s user 0.004s sys 0.002s
I20260807 11:10:12.730885 27964 raft_consensus.cc:482] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:12.731720 27853 ts_tablet_manager.cc:1428] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260807 11:10:12.731429 27964 raft_consensus.cc:504] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:12.738880 27964 leader_election.cc:288] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.739393 27964 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:12.740190 27967 raft_consensus.cc:482] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:12.740937 27967 raft_consensus.cc:504] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:12.743373 27967 leader_election.cc:288] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.744405 27963 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:12.743974 27964 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:12.745505 26980 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 280) took 1860 ms. Trace:
I20260807 11:10:12.744990 27963 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:12.745916 26980 rpcz_store.cc:276] 0807 11:10:10.884844 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:10.885860 (+ 1016us) service_pool.cc:227] Handling call
0807 11:10:10.895560 (+ 9700us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:11.844976 (+949416us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:12.745220 (+900244us) spinlock_profiling.cc:243] Waited 914 ms on lock 0x7b5400470754. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:12.745426 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2101386112}
I20260807 11:10:12.747615 27963 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.748126 27963 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:12.748569 27963 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:12.750484 27963 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:12.751266 25799 raft_consensus.cc:1261] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:12.753103 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:12.753690 25796 raft_consensus.cc:3032] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:12.754004 25796 raft_consensus.cc:729] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:12.754616 27962 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:12.754694 25796 consensus_queue.cc:260] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:12.755234 27962 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:12.756048 25796 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:12.756928 25798 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b8330847089e41b0811e28dc30529234"
dest_uuid: "606d49e053a346e486f48cb31832b815"
mode: GRACEFUL
new_leader_uuid: "4fd6381544574ddca559ce63b9243efc"
from {username='slave'} at 127.0.0.1:58114
I20260807 11:10:12.757560 25798 raft_consensus.cc:595] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Received request to transfer leadership to TS 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:12.758136 25798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:58114 (request call id 6) took 1586 ms. Trace:
I20260807 11:10:12.758512 25798 rpcz_store.cc:276] 0807 11:10:11.171288 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:11.186139 (+ 14851us) service_pool.cc:227] Handling call
0807 11:10:12.758061 (+1571922us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:12.758901 27968 raft_consensus.cc:482] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:12.759464 27968 raft_consensus.cc:504] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:12.761849 27963 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:12.762401 27963 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:12.764518 27963 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.769814 25796 raft_consensus.cc:2454] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:12.774669 27963 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:12.775146 27971 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:12.775298 27963 raft_consensus.cc:504] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:12.775712 27971 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:12.777396 27845 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:12.777509 27963 leader_election.cc:288] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.771368 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:12.777953 27845 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:12.746344 27964 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:12.780190 27845 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.780155 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:12.777932 27971 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:12.781006 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:12.781062 25798 raft_consensus.cc:2454] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 1.
I20260807 11:10:12.781809 25799 raft_consensus.cc:2454] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 1.
I20260807 11:10:12.783898 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:12.786168 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:12.786737 25799 raft_consensus.cc:2454] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 1.
I20260807 11:10:12.795666 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:12.799999 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:12.800809 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:12.800823 27963 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:12.804685 27963 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:12.806659 27963 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.807150 27963 raft_consensus.cc:482] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:12.807642 27963 raft_consensus.cc:504] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:12.800278 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:12.809396 25796 raft_consensus.cc:2454] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:12.800923 27973 raft_consensus.cc:482] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:12.810654 27973 raft_consensus.cc:504] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:12.812065 27845 raft_consensus.cc:482] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:12.812223 27972 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:12.812606 27845 raft_consensus.cc:504] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:11.242398 25944 tablet_copy_service.cc:342] P 4fd6381544574ddca559ce63b9243efc: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-f40bb008c149418989fdcd7c1e67d554 received from {username='slave'} at 127.0.0.1:35276
I20260807 11:10:12.814347 25944 tablet_copy_service.cc:433] P 4fd6381544574ddca559ce63b9243efc: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-f40bb008c149418989fdcd7c1e67d554 on tablet f40bb008c149418989fdcd7c1e67d554 with peer eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:12.815431 25944 rpcz_store.cc:275] Call kudu.tserver.TabletCopyService.EndTabletCopySession from 127.0.0.1:35276 (request call id 60) took 1579 ms. Trace:
I20260807 11:10:12.816911 25944 rpcz_store.cc:276] 0807 11:10:11.236054 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:11.237838 (+ 1784us) service_pool.cc:227] Handling call
0807 11:10:12.815319 (+1577481us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:12.819149 27963 leader_election.cc:288] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.819859 27963 raft_consensus.cc:482] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:12.816224 27971 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:12.820945 27971 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:12.823866 27971 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.821668 27845 leader_election.cc:288] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.827899 25825 tablet_copy_service.cc:342] P 606d49e053a346e486f48cb31832b815: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-cf86a21774c34f33a6540d1dad0de0c4 received from {username='slave'} at 127.0.0.1:59118
I20260807 11:10:12.828516 25825 tablet_copy_service.cc:433] P 606d49e053a346e486f48cb31832b815: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-cf86a21774c34f33a6540d1dad0de0c4 on tablet cf86a21774c34f33a6540d1dad0de0c4 with peer eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:12.830924 26936 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:12.831128 25800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1283) took 1891 ms. Trace:
I20260807 11:10:12.824741 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:12.833791 27757 raft_consensus.cc:2781] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:12.802999 27964 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:12.834635 27964 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:12.834764 25800 rpcz_store.cc:276] 0807 11:10:10.939169 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:10.939302 (+ 133us) service_pool.cc:227] Handling call
0807 11:10:11.114937 (+175635us) spinlock_profiling.cc:243] Waited 8.28 ms on lock 0x7b5800003c98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764bce 000000000034aea9 00007f3fc4590a30 00007f3fc46567b4 00007f3fc4656821 00007f3fc461a2b0 00007f3fc46057f9 00007f3fc46052a2 00007f3fbdceb684
0807 11:10:11.115061 (+ 124us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:11.253968 (+138907us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:11.254011 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:11.254065 (+ 54us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:11.254106 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:12.830775 (+1576669us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b540014dcd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:12.830837 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:12.830992 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":337932928}
I20260807 11:10:12.834556 27757 raft_consensus.cc:686] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:12.837603 27964 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:12.837225 27757 consensus_queue.cc:237] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:12.838265 27964 raft_consensus.cc:482] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:12.838922 27964 raft_consensus.cc:504] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:12.844296 27964 leader_election.cc:288] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.801568 27974 raft_consensus.cc:482] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:12.823127 27973 leader_election.cc:288] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.826640 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:12.845544 25799 raft_consensus.cc:2454] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 1.
I20260807 11:10:12.820544 27963 raft_consensus.cc:504] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:12.831779 27906 consensus_queue.cc:1035] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 11:10:12.832279 26936 leader_election.cc:302] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:12.812973 27972 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:12.851366 26936 leader_election.cc:302] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:12.853163 26936 leader_election.cc:302] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:12.855919 26936 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:12.866896 27963 leader_election.cc:288] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.868731 27972 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:12.849496 27974 raft_consensus.cc:504] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:12.874294 25918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1306) took 1583 ms. Trace:
I20260807 11:10:12.877537 27974 leader_election.cc:288] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.865408 27750 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:12.857635 27733 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:12.874763 25918 rpcz_store.cc:276] 0807 11:10:11.290890 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:12.873205 (+1582315us) service_pool.cc:227] Handling call
0807 11:10:12.873705 (+ 500us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:12.873925 (+ 220us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:12.873961 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:12.873997 (+ 36us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:10:12.874034 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:12.874098 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:12.874226 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:12.858355 27955 raft_consensus.cc:2781] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:12.858120 27957 raft_consensus.cc:2781] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:12.890173 27957 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:12.890689 27957 raft_consensus.cc:3037] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:12.874938 27839 raft_consensus.cc:2781] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:12.894145 27839 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:12.894659 27839 raft_consensus.cc:3037] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:12.898564 27733 raft_consensus.cc:504] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:12.901007 27750 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:12.857223 27907 raft_consensus.cc:2781] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:12.904230 27907 raft_consensus.cc:482] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:12.884833 27821 tablet_copy_client.cc:531] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:10:12.901996 27962 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.887796 27955 raft_consensus.cc:482] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:12.904918 27907 raft_consensus.cc:3037] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:12.908030 27955 raft_consensus.cc:3037] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:12.910962 25926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1313) took 1666 ms. Trace:
I20260807 11:10:12.914500 25926 rpcz_store.cc:276] 0807 11:10:11.241213 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:11.241344 (+ 131us) service_pool.cc:227] Handling call
0807 11:10:12.907148 (+1665804us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:12.907318 (+ 170us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:12.907343 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:12.907388 (+ 45us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:10:12.907414 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:12.907470 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:12.907579 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:12.922695 26936 leader_election.cc:302] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:12.913972 27733 leader_election.cc:288] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.924574 27750 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:12.925302 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1312) took 1688 ms. Trace:
I20260807 11:10:12.940008 25932 rpcz_store.cc:276] 0807 11:10:11.237015 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:11.237145 (+ 130us) service_pool.cc:227] Handling call
0807 11:10:11.237433 (+ 288us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:11.237630 (+ 197us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:11.237659 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:11.237699 (+ 40us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:10:11.237728 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:11.237803 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:12.925236 (+1687433us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:12.929287 27839 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:12.948427 27906 raft_consensus.cc:2781] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:12.948900 27906 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:12.949311 27906 raft_consensus.cc:3037] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:12.953787 25786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 37) took 2011 ms. Trace:
I20260807 11:10:12.954329 25786 rpcz_store.cc:276] 0807 11:10:10.942541 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:10.942702 (+ 161us) service_pool.cc:227] Handling call
0807 11:10:10.943021 (+ 319us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:10.955786 (+ 12765us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:10.955822 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:10.971530 (+ 15708us) log.cc:842] Serialized 26 byte log entry
0807 11:10:11.180576 (+209046us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:10:11.180615 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:11.180694 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:12.953465 (+1772771us) raft_consensus.cc:1671] finished
0807 11:10:12.953496 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:12.953704 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4072,"raft.queue_time_us":12500,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":17108,"thread_start_us":164081,"threads_started":1,"wal-append.queue_time_us":924767,"wal-append.run_cpu_time_us":2032,"wal-append.run_wall_time_us":1016721}
W20260807 11:10:12.965596 25655 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 2.034s user 0.011s sys 0.036s
W20260807 11:10:12.965965 25655 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.034s user 0.011s sys 0.036s
I20260807 11:10:12.970546 27839 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:12.981945 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:12.982139 26980 raft_consensus.cc:1261] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:12.982858 27821 tablet_bootstrap.cc:492] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap starting.
I20260807 11:10:12.979615 27972 raft_consensus.cc:482] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:12.990144 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:12.990805 25786 raft_consensus.cc:3037] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:12.992512 27973 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:12.987573 27962 raft_consensus.cc:482] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:12.993052 27973 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:12.993224 27962 raft_consensus.cc:504] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:12.998432 27962 leader_election.cc:288] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:12.998924 27906 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:12.999645 27973 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:13.001645 25786 raft_consensus.cc:2454] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:10:12.986709 27963 raft_consensus.cc:482] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:13.003353 26936 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:13.003762 27963 raft_consensus.cc:504] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:12.995030 27971 raft_consensus.cc:482] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:13.009171 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:13.013459 26980 raft_consensus.cc:2379] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 3: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:13.014091 27977 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:13.014567 27955 raft_consensus.cc:504] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:13.002737 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:13.000363 27961 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:12.991775 27972 raft_consensus.cc:504] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:13.017907 27968 leader_election.cc:288] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:13.018215 27972 leader_election.cc:288] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:13.018741 27961 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:13.014624 27977 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:13.016323 27907 raft_consensus.cc:504] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:13.028247 27971 raft_consensus.cc:504] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:13.017910 27963 leader_election.cc:288] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:13.040213 27971 leader_election.cc:288] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:13.023244 27962 consensus_queue.cc:1035] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260807 11:10:13.035681 27977 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:13.035750 27907 leader_election.cc:288] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:13.044310 27961 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:13.044798 26979 raft_consensus.cc:2379] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 2: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:13.046675 27971 raft_consensus.cc:979] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815: : Instructing follower 4fd6381544574ddca559ce63b9243efc to start an election
I20260807 11:10:13.047154 27971 raft_consensus.cc:1067] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Signalling peer 4fd6381544574ddca559ce63b9243efc to start an election
I20260807 11:10:13.041111 27906 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:13.045887 27907 raft_consensus.cc:2781] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:13.061565 27957 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:13.066566 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:13.067782 26978 raft_consensus.cc:2379] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 2: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:13.068686 27957 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:13.070449 27955 leader_election.cc:288] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:13.072575 27712 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:13.076181 25919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 57) took 2024 ms. Trace:
I20260807 11:10:13.077317 26984 raft_consensus.cc:1203] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20260807 11:10:13.102875 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 99583.4us
I20260807 11:10:13.079988 25932 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f75da3017fd744719647332808daba49"
dest_uuid: "4fd6381544574ddca559ce63b9243efc"
mode: GRACEFUL
new_leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
from {username='slave'} at 127.0.0.1:34992
I20260807 11:10:13.106181 25932 raft_consensus.cc:595] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Received request to transfer leadership to TS eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:13.106459 25932 raft_consensus.cc:603] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 11:10:13.071985 27849 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:13.104395 25919 rpcz_store.cc:276] 0807 11:10:11.051385 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:11.051510 (+ 125us) service_pool.cc:227] Handling call
0807 11:10:11.051811 (+ 301us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:11.071127 (+ 19316us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:11.071166 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:11.075338 (+ 4172us) log.cc:842] Serialized 26 byte log entry
0807 11:10:11.185581 (+110243us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:10:11.185619 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:11.185696 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:13.075971 (+1890275us) raft_consensus.cc:1671] finished
0807 11:10:13.075992 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:13.076123 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23272,"raft.queue_time_us":22697,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":10102,"thread_start_us":87934,"threads_started":1,"wal-append.queue_time_us":721680,"wal-append.run_cpu_time_us":2815,"wal-append.run_wall_time_us":1108730}
I20260807 11:10:13.071995 27851 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:13.073186 27927 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:13.074101 27907 raft_consensus.cc:686] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:13.073621 27712 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:13.447599 25796 raft_consensus.cc:1261] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:13.118036 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:13.075536 27879 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
W20260807 11:10:13.286477 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 2.04718e+06us
I20260807 11:10:13.627602 25786 raft_consensus.cc:3037] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:13.280036 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:13.629186 25800 raft_consensus.cc:3037] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:13.118865 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:13.631052 25799 raft_consensus.cc:3037] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:13.286584 25932 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d"
dest_uuid: "4fd6381544574ddca559ce63b9243efc"
mode: GRACEFUL
new_leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
from {username='slave'} at 127.0.0.1:34992
I20260807 11:10:13.183630 27926 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:13.076977 27733 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:13.638077 25932 raft_consensus.cc:595] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Received request to transfer leadership to TS eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:13.652082 25786 raft_consensus.cc:2454] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:13.075217 27905 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:13.346127 27990 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:13.275394 27910 raft_consensus.cc:482] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:13.407388 27997 raft_consensus.cc:482] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:13.417672 27998 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:13.183971 27985 raft_consensus.cc:482] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:13.285920 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:13.671280 25798 raft_consensus.cc:3037] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:13.671141 27985 raft_consensus.cc:504] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:13.367841 27996 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:13.672653 25800 raft_consensus.cc:2454] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:13.407569 27750 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:13.673573 27996 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:13.308602 27923 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:13.369323 27999 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:13.689605 25799 raft_consensus.cc:2454] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:13.690716 28017 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:13.693149 28028 raft_consensus.cc:482] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:13.679636 27985 leader_election.cc:288] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:13.676463 27998 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:13.675153 27997 raft_consensus.cc:504] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:13.669586 27910 raft_consensus.cc:504] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:13.781605 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260807 11:10:13.714066 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:13.964505 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:13.965435 25932 raft_consensus.cc:2454] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 1.
I20260807 11:10:13.966615 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:13.967388 25932 raft_consensus.cc:2454] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:13.715235 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:13.970275 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:13.970537 27755 raft_consensus.cc:482] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:13.971657 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:13.971527 27755 raft_consensus.cc:504] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:13.973896 27827 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:13.716969 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:13.974857 27827 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:13.976645 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:13.978404 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:13.979866 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:13.981426 27755 leader_election.cc:288] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:13.981871 25750 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:13.983387 27967 raft_consensus.cc:2781] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:13.983920 27967 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:13.719214 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:13.708892 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 46983.8us
I20260807 11:10:13.722854 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1311) took 2486 ms. Trace:
I20260807 11:10:13.987996 25917 rpcz_store.cc:276] 0807 11:10:11.236650 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:11.236814 (+ 164us) service_pool.cc:227] Handling call
0807 11:10:11.237087 (+ 273us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:11.237264 (+ 177us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:11.237287 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:11.237323 (+ 36us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:11.237355 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:11.237400 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:13.722761 (+2485361us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:13.990061 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:13.717370 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:13.993953 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:13.995563 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:13.996874 26986 raft_consensus.cc:1226] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:13.718266 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:14.000427 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:14.001381 25920 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:14.002633 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:14.004055 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:14.005065 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:14.005645 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:13.717465 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:13.703337 25798 raft_consensus.cc:2454] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:13.716048 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:14.016656 25918 raft_consensus.cc:2454] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:14.018253 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:14.018335 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:14.019181 25919 raft_consensus.cc:2454] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:14.019411 25918 raft_consensus.cc:2454] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:13.734511 25944 tablet_copy_service.cc:342] P 4fd6381544574ddca559ce63b9243efc: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-7be451f12275472a9c2c53877d9d1a0a received from {username='slave'} at 127.0.0.1:35276
I20260807 11:10:13.674065 27750 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:13.661036 27733 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:13.135048 27849 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:13.679260 27923 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:13.662431 27926 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:13.697732 27997 leader_election.cc:288] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:13.697378 27998 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:14.067646 26938 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.707s (SENT) [suppressed 3 similar messages]
I20260807 11:10:13.686233 27999 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:14.070990 27999 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:14.071668 27849 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:13.714464 28029 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:14.072610 27998 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:14.073328 27998 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:13.704214 28028 raft_consensus.cc:504] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:13.703192 27996 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:13.693374 28017 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:13.701676 28024 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:13.677603 27990 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:13.715461 27960 raft_consensus.cc:1050] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:10:14.089108 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:10:14.089512 28017 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:13.809446 27975 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:14.090201 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:10:13.627737 27879 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:13.830185 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:13.667802 27907 consensus_queue.cc:237] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:13.872982 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:13.875053 25655 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:40716 (request call id 244) took 3139 ms. Trace:
I20260807 11:10:13.878463 25654 catalog_manager.cc:5495] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 reported cstate change: term changed from 1 to 2, leader changed from b22d69db45194908ad3d345eba9badd9 (127.23.120.131) to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 2 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 11:10:13.882361 27821 tablet_bootstrap.cc:700] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543: Time spent opening tablet: real 0.206s user 0.005s sys 0.000s
I20260807 11:10:13.876639 28033 consensus_queue.cc:1035] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260807 11:10:13.639294 27851 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:13.919013 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:13.941371 27712 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:13.984429 27967 raft_consensus.cc:3037] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:14.098434 26985 raft_consensus.cc:2379] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 2: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:14.099160 26985 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 292) took 1108 ms. Trace:
I20260807 11:10:14.099483 26985 rpcz_store.cc:276] 0807 11:10:12.990330 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:12.990437 (+ 107us) service_pool.cc:227] Handling call
0807 11:10:13.918418 (+927981us) spinlock_profiling.cc:243] Waited 1.53 ms on lock 0x7b5800000f98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc46061b0 00007f3fc4605a94 00007f3fbdcedc04 00007f3fbdcedb92
0807 11:10:14.099088 (+180670us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3514112}
I20260807 11:10:13.991441 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:13.979888 27827 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:14.014995 27910 leader_election.cc:288] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:14.020386 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:14.020653 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:13.968457 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:14.029027 25796 raft_consensus.cc:1226] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:14.039572 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:14.040917 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:14.041715 25797 raft_consensus.cc:1226] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:13.996351 25921 raft_consensus.cc:2454] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:14.045239 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:14.049633 25944 tablet_copy_service.cc:433] P 4fd6381544574ddca559ce63b9243efc: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-7be451f12275472a9c2c53877d9d1a0a on tablet 7be451f12275472a9c2c53877d9d1a0a with peer eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:14.050130 28025 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:14.052348 27750 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:14.027808 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:14.052966 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:13.876677 26936 leader_election.cc:302] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:13.639359 27927 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:14.057330 27733 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:14.026747 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:14.058120 27882 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:14.058938 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:14.060515 27923 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:13.674587 27905 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:14.027184 28014 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:14.073060 28029 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:14.082732 27998 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:14.092767 25925 raft_consensus.cc:2454] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:14.097505 26987 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 288) took 1113 ms. Trace:
I20260807 11:10:14.086278 28024 raft_consensus.cc:504] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:14.090517 27975 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:14.152149 25918 raft_consensus.cc:2454] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 1.
I20260807 11:10:14.236006 26987 rpcz_store.cc:276] 0807 11:10:12.984498 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:12.984662 (+ 164us) service_pool.cc:227] Handling call
0807 11:10:13.872477 (+887815us) spinlock_profiling.cc:243] Waited 2.6 ms on lock 0x7b5800000f98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc46061b0 00007f3fc4605a94 00007f3fbdcedc04 00007f3fbdcedb92
0807 11:10:14.097403 (+224926us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5972224}
I20260807 11:10:14.139715 25799 raft_consensus.cc:1226] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:14.239259 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:14.201283 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:14.241995 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:14.225282 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 46284.1us
I20260807 11:10:14.157581 27712 raft_consensus.cc:482] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:14.251343 25931 raft_consensus.cc:2454] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
W20260807 11:10:14.227445 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 130377us
I20260807 11:10:14.252849 25919 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:14.253458 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:14.103899 28017 raft_consensus.cc:482] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:14.255537 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:14.253824 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:14.257695 25918 raft_consensus.cc:2454] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 1.
I20260807 11:10:14.204917 28045 raft_consensus.cc:482] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
W20260807 11:10:14.254439 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 378260us
I20260807 11:10:14.174292 27926 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:14.261413 28033 raft_consensus.cc:2781] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:14.261700 28039 raft_consensus.cc:482] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:14.262094 28033 raft_consensus.cc:686] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:14.173130 28028 leader_election.cc:288] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:14.266021 28024 leader_election.cc:288] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:14.250274 27755 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:14.264926 25801 raft_consensus.cc:2454] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:14.251785 28014 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:14.284790 27712 raft_consensus.cc:504] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:14.285995 25926 raft_consensus.cc:2454] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:14.286651 28014 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:14.217015 27849 consensus_queue.cc:1046] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260807 11:10:14.290021 27990 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:14.290278 27903 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:14.290114 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:14.291066 27927 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:14.291113 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:14.262492 28039 raft_consensus.cc:504] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:14.293972 25806 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:14.299099 27915 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:14.299753 26987 raft_consensus.cc:2454] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:14.299883 28029 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:14.290129 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:14.286918 27712 leader_election.cc:288] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:14.226780 27960 consensus_queue.cc:237] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:14.264424 27882 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:14.265682 28033 consensus_queue.cc:237] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:14.456444 25796 raft_consensus.cc:1226] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:14.304608 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:14.318285 26936 leader_election.cc:302] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:14.391258 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:14.293428 27875 raft_consensus.cc:482] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:14.439536 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:14.459786 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 49176.5us
I20260807 11:10:14.303442 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:14.375591 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:14.431694 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:14.321202 26985 raft_consensus.cc:1203] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:10:14.483013 25799 raft_consensus.cc:2454] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:14.489812 28036 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:14.489831 25916 raft_consensus.cc:2454] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 1.
I20260807 11:10:14.490496 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:14.295403 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:14.493710 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:14.495155 26980 raft_consensus.cc:1226] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:14.491837 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:14.385085 27926 raft_consensus.cc:482] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:14.298682 28017 raft_consensus.cc:504] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:14.325670 27851 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:14.326035 26979 raft_consensus.cc:2454] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:14.497709 25917 raft_consensus.cc:2454] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 1.
I20260807 11:10:14.391032 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:14.284395 27827 raft_consensus.cc:482] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:14.500806 28017 leader_election.cc:288] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:14.454082 27727 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:14.496565 27926 raft_consensus.cc:504] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:14.319806 25750 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:14.391517 28034 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:14.438139 28039 leader_election.cc:288] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:14.455601 25926 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8330847089e41b0811e28dc30529234"
dest_uuid: "4fd6381544574ddca559ce63b9243efc"
from {username='slave'} at 127.0.0.1:39742
W20260807 11:10:14.510113 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 66606.1us
I20260807 11:10:14.510950 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:14.321482 27755 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:14.491037 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:14.464772 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 44550.1us
I20260807 11:10:14.545822 25926 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:10:14.550127 25926 raft_consensus.cc:3037] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:14.304431 28025 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:14.506347 28042 raft_consensus.cc:482] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:14.485000 27967 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:14.564062 25786 raft_consensus.cc:1226] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:14.490480 28036 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:14.571583 28036 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:14.572279 28036 raft_consensus.cc:482] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:14.424314 28045 raft_consensus.cc:504] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:14.573614 25920 raft_consensus.cc:2454] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:14.575001 28045 leader_election.cc:288] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:14.567005 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 295119us
I20260807 11:10:14.549149 28039 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:14.513127 27975 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:14.325477 27903 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:14.481727 27915 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:14.595175 25799 raft_consensus.cc:2454] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:14.510547 27876 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:14.551188 27879 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:14.599033 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:14.601272 25808 raft_consensus.cc:1226] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:14.604314 26936 leader_election.cc:302] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:14.606449 27963 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:14.092571 25655 rpcz_store.cc:276] 0807 11:10:10.752203 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:10.774392 (+ 22189us) service_pool.cc:227] Handling call
0807 11:10:13.874781 (+3100389us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 11:10:12.822831 (+ 0us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b5400003128. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc5be5e91 00007f3fc5be4afc 00007f3fc30e77cc 00007f3fc30d3603 00007f3fc30d32b8 00007f3fc30d7ba1 00007f3fc30d7b59 00007f3fc30d7ae9 00007f3fc30d7ab1 00007f3fc30d6dad 00007f3fc5a00204
0807 11:10:12.822966 (+ 135us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 11:10:12.823734 (+ 768us) write_op.cc:432] Acquiring schema lock in shared mode
0807 11:10:12.823778 (+ 44us) write_op.cc:435] Acquired schema lock
0807 11:10:12.823802 (+ 24us) tablet.cc:660] Decoding operations
0807 11:10:12.824327 (+ 525us) write_op.cc:620] Acquiring the partition lock for write op
0807 11:10:12.824405 (+ 78us) write_op.cc:641] Partition lock acquired for write op
0807 11:10:12.824454 (+ 49us) tablet.cc:683] Acquiring locks for 1 operations
0807 11:10:12.825080 (+ 626us) tablet.cc:699] Row locks acquired
0807 11:10:12.825102 (+ 22us) write_op.cc:260] PREPARE: finished
0807 11:10:12.825286 (+ 184us) write_op.cc:270] Start()
0807 11:10:12.825693 (+ 407us) write_op.cc:276] Timestamp: P: 1786101012825232 usec, L: 0
0807 11:10:12.825871 (+ 178us) op_driver.cc:331] REPLICATION: starting
0807 11:10:12.826740 (+ 869us) log.cc:842] Serialized 539 byte log entry
0807 11:10:13.402327 (+575587us) op_driver.cc:444] REPLICATION: finished
0807 11:10:13.668276 (+265949us) write_op.cc:301] APPLY: starting
0807 11:10:13.668364 (+ 88us) tablet.cc:1365] starting BulkCheckPresence
0807 11:10:13.668475 (+ 111us) tablet.cc:1368] finished BulkCheckPresence
0807 11:10:13.668498 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 11:10:13.668749 (+ 251us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 11:10:13.668919 (+ 170us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0807 11:10:13.668966 (+ 47us) write_op.cc:312] APPLY: finished
0807 11:10:13.669190 (+ 224us) log.cc:842] Serialized 23 byte log entry
0807 11:10:13.669348 (+ 158us) write_op.cc:489] Releasing partition, row and schema locks
0807 11:10:13.669433 (+ 85us) write_op.cc:454] Released schema lock
0807 11:10:13.669470 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0807 11:10:13.669752 (+ 282us) write_op.cc:489] Releasing partition, row and schema locks
0807 11:10:13.669783 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":119686,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":260947,"apply.run_cpu_time_us":1768,"apply.run_wall_time_us":1769,"num_ops":1,"prepare.queue_time_us":885043,"prepare.run_cpu_time_us":63109,"prepare.run_wall_time_us":1160988,"raft.queue_time_us":347856,"raft.run_cpu_time_us":6775,"raft.run_wall_time_us":72364,"replication_time_us":576282,"spinlock_wait_cycles":2306096512,"thread_start_us":2598910,"threads_started":4,"wal-append.queue_time_us":160348}]]}
I20260807 11:10:14.579704 28037 raft_consensus.cc:482] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:14.561079 27827 raft_consensus.cc:504] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:14.593437 27975 raft_consensus.cc:2726] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:14.613734 27907 raft_consensus.cc:2781] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:14.613909 27882 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:14.614172 28052 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:14.566509 27875 raft_consensus.cc:504] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:14.614634 27876 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:14.620370 27963 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:14.594661 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:14.611449 27905 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:14.633812 25869 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:14.615149 27880 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:14.641004 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:14.644023 26936 leader_election.cc:302] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:14.594964 28034 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:14.631628 28017 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:14.659475 27975 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:14.659678 28017 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:14.631531 27963 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:14.635030 28030 consensus_queue.cc:1046] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260807 11:10:14.625777 28039 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:14.658150 28054 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:14.658425 26036 raft_consensus.cc:1261] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:14.658881 27915 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:14.614709 28042 raft_consensus.cc:504] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:14.623165 28036 raft_consensus.cc:504] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:14.660311 27975 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:14.611384 27727 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:14.688743 26079 debug-util.cc:398] Leaking SignalData structure 0x7b08000f0080 after lost signal to thread 24038
I20260807 11:10:14.688910 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.689832 26034 raft_consensus.cc:2454] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:14.689977 28042 leader_election.cc:288] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:10:14.690387 26079 debug-util.cc:398] Leaking SignalData structure 0x7b080019bea0 after lost signal to thread 25705
I20260807 11:10:14.690582 28042 raft_consensus.cc:482] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:14.691243 28042 raft_consensus.cc:504] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:14.692428 26079 debug-util.cc:398] Leaking SignalData structure 0x7b0800293120 after lost signal to thread 25850
W20260807 11:10:14.725553 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 91805us
I20260807 11:10:14.684512 28028 raft_consensus.cc:2726] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:14.693514 28042 leader_election.cc:288] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:14.691521 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.724231 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.658716 28055 raft_consensus.cc:482] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:14.696548 28039 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:14.695454 28036 leader_election.cc:288] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:14.693239 28058 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:14.660938 28037 raft_consensus.cc:504] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:14.675274 27915 raft_consensus.cc:482] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:14.723080 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.696997 28054 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:14.657235 28052 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:14.677230 28051 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
W20260807 11:10:14.728348 26079 debug-util.cc:398] Leaking SignalData structure 0x7b08003553c0 after lost signal to thread 25851
I20260807 11:10:14.735556 25920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1347) took 1009 ms. Trace:
I20260807 11:10:14.729965 27926 leader_election.cc:288] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:14.748425 28036 raft_consensus.cc:482] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:14.748600 28039 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:14.749310 28042 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
W20260807 11:10:14.749472 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 429846us
I20260807 11:10:14.735311 27907 raft_consensus.cc:686] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:14.731192 27755 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:14.681104 27880 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:14.755542 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.756031 27876 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:14.765689 27996 raft_consensus.cc:2781] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:14.786851 26042 raft_consensus.cc:2454] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 1.
I20260807 11:10:14.763455 27827 leader_election.cc:288] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:14.763835 28056 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:14.742702 28025 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:14.766630 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.792032 26034 raft_consensus.cc:2454] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:14.767505 25920 rpcz_store.cc:276] 0807 11:10:13.726336 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:14.058583 (+332247us) service_pool.cc:227] Handling call
0807 11:10:14.735335 (+676752us) spinlock_profiling.cc:243] Waited 334 ms on lock 0x7b54000532d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:14.735484 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":769320320}
I20260807 11:10:14.779620 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:14.780823 25919 consensus_meta.cc:287] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.513s user 0.011s sys 0.000s
I20260807 11:10:14.793365 25919 raft_consensus.cc:2454] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:14.793900 25919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35276 (request call id 66) took 1063 ms. Trace:
I20260807 11:10:14.761545 27967 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:14.764726 28034 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:14.756357 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.785485 26033 raft_consensus.cc:2454] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 1.
I20260807 11:10:14.759781 28058 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:14.764865 27875 leader_election.cc:288] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:14.765828 27990 raft_consensus.cc:3032] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:14.775743 25869 leader_election.cc:302] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:14.750765 27905 consensus_queue.cc:1046] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.036s
I20260807 11:10:14.794298 25919 rpcz_store.cc:276] 0807 11:10:13.730944 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:14.167368 (+436424us) service_pool.cc:227] Handling call
0807 11:10:14.793844 (+626476us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:14.805866 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.796376 28059 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:14.796551 28036 raft_consensus.cc:504] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:14.796633 27907 consensus_queue.cc:237] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:14.797245 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.790488 27915 raft_consensus.cc:504] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:14.807731 25750 leader_election.cc:302] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:14.764890 28055 raft_consensus.cc:504] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:14.798653 28037 leader_election.cc:288] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:14.798828 28061 raft_consensus.cc:2781] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election won for term 2
W20260807 11:10:14.802595 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 175804us
I20260807 11:10:14.804886 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.791160 27755 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:14.809221 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:14.814616 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 63695.7us
I20260807 11:10:14.817035 26034 raft_consensus.cc:2454] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:14.804951 27990 raft_consensus.cc:729] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:14.819013 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.819119 28065 raft_consensus.cc:482] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:14.805073 27996 raft_consensus.cc:686] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:14.791495 28039 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:14.819162 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.809862 27903 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:14.822197 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.825824 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.826555 26038 raft_consensus.cc:2454] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 1.
I20260807 11:10:14.824682 26039 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:14.804450 28042 raft_consensus.cc:504] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:14.812319 27727 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:14.848984 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.849192 25869 leader_election.cc:302] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:14.849692 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.848963 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:14.823729 27905 raft_consensus.cc:2781] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:14.853117 26039 raft_consensus.cc:2454] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:14.817793 27967 raft_consensus.cc:2781] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:14.850929 27990 consensus_queue.cc:260] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:14.852136 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.855372 28064 consensus_queue.cc:1046] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260807 11:10:14.857755 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:14.804674 28051 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:14.822335 27975 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:14.852052 27903 consensus_queue.cc:1046] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260807 11:10:14.871219 27990 raft_consensus.cc:3037] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
W20260807 11:10:14.872035 26079 debug-util.cc:398] Leaking SignalData structure 0x7b080030ee60 after lost signal to thread 25964
I20260807 11:10:14.874004 27880 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:14.867168 27882 raft_consensus.cc:2781] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:14.886059 27882 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:14.886214 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 64) took 1156 ms. Trace:
I20260807 11:10:14.886667 27882 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:14.888574 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:14.874708 28052 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:14.874828 25869 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:14.811115 28056 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:14.876959 27755 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:14.856889 27905 raft_consensus.cc:482] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:14.881446 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.859855 27996 consensus_queue.cc:237] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:14.852312 28065 raft_consensus.cc:504] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:14.883143 28036 leader_election.cc:288] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:14.884284 28017 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:14.859442 27967 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:14.887627 28039 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:14.886749 25921 rpcz_store.cc:276] 0807 11:10:13.730164 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:14.495596 (+765432us) service_pool.cc:227] Handling call
0807 11:10:14.885316 (+389720us) spinlock_profiling.cc:243] Waited 6.88 ms on lock 0x7b58002e0e98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc461a2b0 00007f3fc46057f9 00007f3fc46052a2 00007f3fbdceb684
0807 11:10:14.885562 (+ 246us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:14.885773 (+ 211us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:14.885803 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:14.885859 (+ 56us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:14.885894 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:14.885975 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:14.886114 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15827840}
I20260807 11:10:14.868232 28054 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:14.893548 27876 raft_consensus.cc:3032] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:14.906956 27905 raft_consensus.cc:3037] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:14.917341 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.909538 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.914018 26034 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:14.913595 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.910454 27755 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:14.897359 28058 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:14.908560 28061 raft_consensus.cc:686] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:14.917695 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.894219 28056 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:14.894790 27733 raft_consensus.cc:2781] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:14.856307 28059 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:14.895582 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.921736 26036 raft_consensus.cc:2454] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:14.920729 26042 raft_consensus.cc:2454] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 1.
I20260807 11:10:14.926442 26038 raft_consensus.cc:2454] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:14.896555 27903 raft_consensus.cc:3032] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:14.928859 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.929565 26039 raft_consensus.cc:2454] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:14.897279 28055 leader_election.cc:288] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:14.900863 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.902665 28030 raft_consensus.cc:3032] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:14.901127 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.898602 25926 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:14.904337 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 80934.3us
I20260807 11:10:14.897895 25750 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:14.935945 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 69280.6us
I20260807 11:10:14.936239 25926 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:14.936935 25926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:39742 (request call id 1349) took 1209 ms. Trace:
I20260807 11:10:14.937084 25750 leader_election.cc:302] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:14.937351 25926 rpcz_store.cc:276] 0807 11:10:13.727099 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:14.455323 (+728224us) service_pool.cc:227] Handling call
0807 11:10:14.936802 (+481479us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:14.924985 27733 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:14.938501 27733 raft_consensus.cc:3037] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:14.921950 27876 raft_consensus.cc:729] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:14.929247 27903 raft_consensus.cc:729] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:14.942027 27876 consensus_queue.cc:260] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:14.942407 27903 consensus_queue.cc:260] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:14.943790 27876 raft_consensus.cc:3037] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:14.943953 27903 raft_consensus.cc:3037] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:14.945039 27963 raft_consensus.cc:2781] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:14.945564 27963 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:14.946977 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.895004 25869 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:14.928547 27967 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
W20260807 11:10:14.910655 26079 debug-util.cc:398] Leaking SignalData structure 0x7b0800272480 after lost signal to thread 26083
I20260807 11:10:14.931491 28068 consensus_queue.cc:1035] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260807 11:10:14.927956 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.923484 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.948828 27975 raft_consensus.cc:2781] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:14.953444 27975 raft_consensus.cc:482] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:14.953874 27975 raft_consensus.cc:3037] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:14.953787 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.921908 28051 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:10:14.955991 28053 log.cc:924] Time spent T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.173s user 0.002s sys 0.000s
I20260807 11:10:14.949357 27963 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:14.959308 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.959867 26042 raft_consensus.cc:2454] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 1.
I20260807 11:10:14.962742 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1353) took 3768 ms. Trace:
I20260807 11:10:14.963129 26037 rpcz_store.cc:276] 0807 11:10:11.237018 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:11.285585 (+ 48567us) service_pool.cc:227] Handling call
0807 11:10:11.286079 (+ 494us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:14.083639 (+2797560us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:14.084830 (+ 1191us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:14.178024 (+ 93194us) log.cc:842] Serialized 26 byte log entry
0807 11:10:14.525944 (+347920us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:10:14.525996 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:14.559495 (+ 33499us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:14.955357 (+395862us) spinlock_profiling.cc:243] Waited 1.84 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:10:14.957093 (+ 1736us) raft_consensus.cc:1671] finished
0807 11:10:14.957119 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:14.957451 (+ 332us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2678025,"raft.queue_time_us":2692028,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":247,"spinlock_wait_cycles":4237568,"thread_start_us":444959,"threads_started":2,"wal-append.queue_time_us":551139}
I20260807 11:10:14.964802 27915 leader_election.cc:288] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:10:14.966012 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 52186.3us
I20260807 11:10:14.966023 26936 leader_election.cc:302] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:14.933977 28030 raft_consensus.cc:729] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:14.946739 28042 leader_election.cc:288] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:14.968047 25981 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
W20260807 11:10:14.968555 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 73639.9us
I20260807 11:10:14.974418 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:14.974742 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:14.975852 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:14.977283 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:14.977375 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:14.977473 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:14.983352 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:14.984170 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.985052 25750 leader_election.cc:302] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:14.983253 25797 raft_consensus.cc:2379] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 2: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:14.984019 25806 raft_consensus.cc:2454] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:14.934096 28052 consensus_queue.cc:1046] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20260807 11:10:14.967442 28051 consensus_queue.cc:1046] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:14.929052 26938 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:14.987942 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:14.992336 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:14.978670 28017 raft_consensus.cc:2781] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:14.979862 28068 raft_consensus.cc:2781] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260807 11:10:14.995368 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 45103.8us
I20260807 11:10:14.992540 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:14.996598 25981 leader_election.cc:302] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:14.998844 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:14.923179 28061 consensus_queue.cc:237] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:15.001879 25869 leader_election.cc:302] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:15.005810 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.009112 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:15.009871 28030 consensus_queue.cc:260] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:15.025607 27712 raft_consensus.cc:2781] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:14.995800 28017 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:15.011735 25786 raft_consensus.cc:2454] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:15.025897 26936 leader_election.cc:302] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:15.027043 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.025426 25869 leader_election.cc:302] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:15.028308 27927 raft_consensus.cc:2781] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:14.986728 28042 raft_consensus.cc:2726] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:15.028884 27927 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:15.029389 27927 raft_consensus.cc:3037] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:15.029649 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:15.017086 26985 raft_consensus.cc:2454] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:15.025039 25803 raft_consensus.cc:2454] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:15.025375 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:15.031617 25869 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:15.026151 28017 raft_consensus.cc:3037] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:15.019793 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:15.018462 27821 tablet_bootstrap.cc:492] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:10:15.034540 28056 raft_consensus.cc:2726] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:15.038921 26976 raft_consensus.cc:2454] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:15.034953 26987 raft_consensus.cc:1203] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:10:15.034950 28073 raft_consensus.cc:2781] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:15.039986 28073 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:15.040135 25808 raft_consensus.cc:3032] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:15.040490 25808 raft_consensus.cc:729] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:15.040813 27817 raft_consensus.cc:2781] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:15.041186 27817 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:15.041565 27817 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:15.041343 25808 consensus_queue.cc:260] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:15.041476 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:10:15.002827 26079 debug-util.cc:398] Leaking SignalData structure 0x7b080035e580 after lost signal to thread 27031
I20260807 11:10:14.996393 28068 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:15.051318 25869 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:15.054986 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:15.026053 27712 raft_consensus.cc:482] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:15.068153 25808 raft_consensus.cc:3037] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
W20260807 11:10:15.056615 26079 debug-util.cc:398] Leaking SignalData structure 0x7b080035e560 after lost signal to thread 27033
I20260807 11:10:15.040401 28073 raft_consensus.cc:3037] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:15.033910 26980 raft_consensus.cc:2454] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:15.076406 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:15.036283 28059 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:15.083153 28078 raft_consensus.cc:2781] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:15.039809 28065 leader_election.cc:288] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:15.037822 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 107095us
I20260807 11:10:15.085865 28055 raft_consensus.cc:3032] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:15.061348 28068 raft_consensus.cc:3037] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:15.064847 27712 raft_consensus.cc:3037] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:15.116183 28078 raft_consensus.cc:482] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
W20260807 11:10:15.115325 26079 debug-util.cc:398] Leaking SignalData structure 0x7b0800353080 after lost signal to thread 27819
I20260807 11:10:15.036628 28072 raft_consensus.cc:2781] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:15.126330 28078 raft_consensus.cc:3037] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:15.147863 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.158704 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.121762 28055 raft_consensus.cc:729] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:15.162446 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.166507 26936 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:15.172847 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.172938 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:15.174974 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 123748us
W20260807 11:10:15.172356 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 213633us
I20260807 11:10:15.177518 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:15.178378 26042 raft_consensus.cc:2454] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:15.140033 28072 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:15.183233 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.164582 27821 tablet_bootstrap.cc:492] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543: Bootstrap complete.
I20260807 11:10:15.190722 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:15.193365 26936 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:15.176964 28025 raft_consensus.cc:2726] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:15.199788 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:15.199939 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.201188 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.200208 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:15.203125 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.203351 25750 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:15.203266 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.208408 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.208160 27882 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:15.187423 28072 raft_consensus.cc:3037] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:15.195202 28067 raft_consensus.cc:2781] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:15.216346 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:15.217667 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:15.218070 25786 raft_consensus.cc:2396] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260807 11:10:15.222147 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:15.224452 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.209906 28075 raft_consensus.cc:2726] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:15.230217 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.230836 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.231283 25750 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
W20260807 11:10:15.231541 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 51469.9us
W20260807 11:10:15.233139 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 46149.6us
I20260807 11:10:15.235913 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.236078 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.221917 27821 ts_tablet_manager.cc:1397] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543: Time spent bootstrapping tablet: real 2.236s user 0.049s sys 0.015s
I20260807 11:10:15.236734 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.237610 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:15.217278 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.227319 27960 raft_consensus.cc:2781] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:15.240002 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:15.242170 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.244642 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.216777 28067 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:15.251060 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.253850 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.254664 26936 leader_election.cc:302] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:15.266826 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.267429 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:15.267765 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.270352 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.276057 25750 leader_election.cc:302] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:15.279711 28030 raft_consensus.cc:3037] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:15.281051 28075 raft_consensus.cc:2781] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:15.281133 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:15.281641 28075 raft_consensus.cc:482] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:15.282200 28075 raft_consensus.cc:3037] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:15.285636 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:15.266101 28067 raft_consensus.cc:3037] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:15.256875 27960 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:15.271616 28039 raft_consensus.cc:3032] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:15.250993 28079 raft_consensus.cc:2781] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:15.301258 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:15.309149 26936 leader_election.cc:302] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:15.309754 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:15.302961 27960 raft_consensus.cc:3037] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:15.250170 28055 consensus_queue.cc:260] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:15.316144 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:15.302376 28039 raft_consensus.cc:729] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:15.299224 28065 raft_consensus.cc:2781] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:15.330440 26041 raft_consensus.cc:2454] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:15.331787 27905 raft_consensus.cc:504] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:15.333133 27733 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:15.303589 28079 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:15.359100 27905 leader_election.cc:288] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:15.359977 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:15.367164 25916 raft_consensus.cc:1226] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:15.368160 26041 raft_consensus.cc:2454] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:15.371804 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:15.385860 28039 consensus_queue.cc:260] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:15.387907 28039 raft_consensus.cc:3037] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:15.322242 27882 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:15.403676 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:15.339192 28033 raft_consensus.cc:2781] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:15.316610 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:15.338966 28065 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:15.429554 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:15.429998 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:15.430307 26041 raft_consensus.cc:2454] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
W20260807 11:10:15.411445 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 81578.8us
W20260807 11:10:15.360918 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 59275.9us
I20260807 11:10:15.371755 28079 raft_consensus.cc:3037] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:15.435009 25808 raft_consensus.cc:2454] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:15.458372 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:15.422612 28033 raft_consensus.cc:482] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
W20260807 11:10:15.461807 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 151386us
I20260807 11:10:15.456712 25750 leader_election.cc:302] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:15.429559 28065 raft_consensus.cc:3037] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:15.434340 25931 raft_consensus.cc:1226] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:15.476807 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:15.477655 26041 raft_consensus.cc:2454] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:15.484254 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:15.464745 28033 raft_consensus.cc:3037] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:15.427171 27967 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:15.490999 27733 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:15.495414 26980 raft_consensus.cc:1226] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:15.529474 25808 raft_consensus.cc:1261] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260807 11:10:15.533696 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 77111us
I20260807 11:10:15.537422 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:15.487077 28017 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:15.581629 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:15.611725 25806 raft_consensus.cc:1226] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:15.580731 27967 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:15.552572 27903 consensus_meta.cc:287] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.608s user 0.019s sys 0.007s
I20260807 11:10:15.616194 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:15.614619 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:15.621343 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:15.623641 28017 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:15.624979 26037 raft_consensus.cc:2454] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:15.617775 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:15.616979 28090 log.cc:924] Time spent T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.207s user 0.001s sys 0.000s
I20260807 11:10:15.609659 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:15.637588 26984 raft_consensus.cc:1226] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:15.626912 26979 raft_consensus.cc:1226] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
W20260807 11:10:15.645180 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 72234.1us
I20260807 11:10:15.632162 27967 raft_consensus.cc:2781] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:15.661653 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:15.662029 27967 raft_consensus.cc:482] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:15.662535 27967 raft_consensus.cc:3037] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:15.663082 27821 raft_consensus.cc:348] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:15.664034 27821 raft_consensus.cc:729] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Initialized, Role: LEARNER
W20260807 11:10:15.646842 27975 consensus_meta.cc:287] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.683s user 0.006s sys 0.015s
I20260807 11:10:15.635622 25918 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:10:15.666853 27821 consensus_queue.cc:260] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:15.669199 28072 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:15.636081 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 154003us
I20260807 11:10:15.624214 28075 raft_consensus.cc:504] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:15.674419 26985 raft_consensus.cc:1226] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:15.676393 26938 leader_election.cc:302] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
W20260807 11:10:15.677644 28095 log.cc:924] Time spent T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260807 11:10:15.637369 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:15.664836 28055 raft_consensus.cc:3037] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:15.693766 25654 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:40714 (request call id 224) took 4826 ms. Trace:
I20260807 11:10:15.678223 25658 catalog_manager.cc:5495] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: term changed from 1 to 2, leader changed from 4fd6381544574ddca559ce63b9243efc (127.23.120.130) to eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132). New cstate: current_term: 2 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:10:15.693578 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:15.699090 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 460300us
I20260807 11:10:15.696727 25750 leader_election.cc:302] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:15.701335 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:10:15.704944 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 117126us
W20260807 11:10:15.700892 27990 consensus_meta.cc:287] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.803s user 0.009s sys 0.013s
I20260807 11:10:15.689198 27907 raft_consensus.cc:2781] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:15.707610 25658 catalog_manager.cc:5495] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: term changed from 1 to 2, leader changed from 4fd6381544574ddca559ce63b9243efc (127.23.120.130) to eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132). New cstate: current_term: 2 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260807 11:10:15.693545 28085 log.cc:924] Time spent T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.243s user 0.001s sys 0.000s
I20260807 11:10:15.708674 25869 leader_election.cc:302] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:15.682552 28072 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:15.693686 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:15.675056 27975 raft_consensus.cc:504] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:15.833133 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RunLeaderElection -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 87981.5us
W20260807 11:10:15.777132 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 63989.2us
W20260807 11:10:15.698933 27927 consensus_meta.cc:287] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.669s user 0.016s sys 0.005s
I20260807 11:10:15.777002 25658 catalog_manager.cc:5495] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: term changed from 1 to 2, leader changed from 606d49e053a346e486f48cb31832b815 (127.23.120.129) to eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132). New cstate: current_term: 2 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:10:15.776665 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:15.841063 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 132458us
I20260807 11:10:15.845342 25869 leader_election.cc:302] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:15.803216 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:15.844568 25918 raft_consensus.cc:2454] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:10:15.753671 28102 raft_consensus.cc:2781] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:15.851455 28102 raft_consensus.cc:482] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:15.851640 28075 leader_election.cc:288] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:15.851852 28102 raft_consensus.cc:3037] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:15.852999 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:15.853804 26041 raft_consensus.cc:2454] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:15.824018 27821 ts_tablet_manager.cc:1428] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543: Time spent starting tablet: real 0.554s user 0.013s sys 0.012s
I20260807 11:10:15.708451 26976 raft_consensus.cc:1226] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:15.858757 25808 raft_consensus.cc:1226] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:15.695482 25654 rpcz_store.cc:276] 0807 11:10:10.867367 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:10.867495 (+ 128us) service_pool.cc:227] Handling call
0807 11:10:15.693669 (+4826174us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 11:10:14.431072 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 11:10:14.609778 (+178706us) write_op.cc:432] Acquiring schema lock in shared mode
0807 11:10:14.618201 (+ 8423us) write_op.cc:435] Acquired schema lock
0807 11:10:14.620328 (+ 2127us) tablet.cc:660] Decoding operations
0807 11:10:14.684076 (+ 63748us) write_op.cc:620] Acquiring the partition lock for write op
0807 11:10:14.714914 (+ 30838us) write_op.cc:641] Partition lock acquired for write op
0807 11:10:14.721602 (+ 6688us) tablet.cc:683] Acquiring locks for 1 operations
0807 11:10:14.722002 (+ 400us) tablet.cc:699] Row locks acquired
0807 11:10:14.722040 (+ 38us) write_op.cc:260] PREPARE: finished
0807 11:10:14.722252 (+ 212us) write_op.cc:270] Start()
0807 11:10:14.722451 (+ 199us) write_op.cc:276] Timestamp: P: 1786101014722170 usec, L: 0
0807 11:10:14.722492 (+ 41us) op_driver.cc:331] REPLICATION: starting
0807 11:10:14.723273 (+ 781us) log.cc:842] Serialized 539 byte log entry
0807 11:10:15.084339 (+361066us) op_driver.cc:444] REPLICATION: finished
0807 11:10:15.306358 (+222019us) write_op.cc:301] APPLY: starting
0807 11:10:15.314953 (+ 8595us) tablet.cc:1365] starting BulkCheckPresence
0807 11:10:15.363945 (+ 48992us) tablet.cc:1368] finished BulkCheckPresence
0807 11:10:15.432062 (+ 68117us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 11:10:15.481349 (+ 49287us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 11:10:15.530274 (+ 48925us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0807 11:10:15.550125 (+ 19851us) write_op.cc:312] APPLY: finished
0807 11:10:15.606645 (+ 56520us) log.cc:842] Serialized 23 byte log entry
0807 11:10:15.631091 (+ 24446us) write_op.cc:489] Releasing partition, row and schema locks
0807 11:10:15.634783 (+ 3692us) write_op.cc:454] Released schema lock
0807 11:10:15.654348 (+ 19565us) write_op.cc:341] FINISH: Updating metrics
0807 11:10:15.669230 (+ 14882us) write_op.cc:489] Releasing partition, row and schema locks
0807 11:10:15.669261 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":63303,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":204769,"apply.run_cpu_time_us":3699,"apply.run_wall_time_us":379703,"num_ops":1,"prepare.queue_time_us":143548,"prepare.run_cpu_time_us":27893,"prepare.run_wall_time_us":432177,"raft.queue_time_us":100549,"raft.run_cpu_time_us":13807,"raft.run_wall_time_us":140328,"replication_time_us":361748,"thread_start_us":298593,"threads_started":4,"wal-append.queue_time_us":121897}]]}
I20260807 11:10:15.859745 25806 raft_consensus.cc:1261] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260807 11:10:15.741226 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 64912.4us
I20260807 11:10:15.754741 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:10:15.849154 28078 consensus_meta.cc:287] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.680s user 0.012s sys 0.007s
I20260807 11:10:15.847582 28092 raft_consensus.cc:2781] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:15.865684 25796 raft_consensus.cc:1261] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:15.713596 27907 raft_consensus.cc:482] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:15.868270 25981 leader_election.cc:302] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:15.917430 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 55010us
W20260807 11:10:15.917641 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 49499.5us
I20260807 11:10:15.869697 27975 leader_election.cc:288] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:15.869964 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:15.870704 28092 raft_consensus.cc:482] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:15.849443 27882 raft_consensus.cc:2781] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:15.871516 25869 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
W20260807 11:10:15.871568 27960 consensus_meta.cc:287] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.527s user 0.006s sys 0.005s
W20260807 11:10:15.919485 27963 consensus_meta.cc:287] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.959s user 0.004s sys 0.014s
W20260807 11:10:15.864113 27821 ts_tablet_manager.cc:1460] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543: Tablet startup took 624ms
I20260807 11:10:15.918361 26938 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:15.929946 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:15.925055 28039 raft_consensus.cc:2726] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:15.926847 27907 raft_consensus.cc:3037] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
W20260807 11:10:15.874212 28068 consensus_meta.cc:287] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.750s user 0.008s sys 0.008s
W20260807 11:10:15.921880 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 62692.6us
I20260807 11:10:15.927433 25918 raft_consensus.cc:1226] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:15.818275 28058 consensus_queue.cc:1046] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.107s
I20260807 11:10:15.867017 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:15.943511 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 74964.6us
I20260807 11:10:15.740448 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:15.842238 27967 raft_consensus.cc:504] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:15.909386 28075 consensus_queue.cc:1035] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260807 11:10:15.961418 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:15.859200 28061 consensus_queue.cc:1046] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.080s
I20260807 11:10:15.934489 28092 raft_consensus.cc:3037] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:15.849560 27927 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:15.937186 27882 raft_consensus.cc:482] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:15.859458 28072 consensus_queue.cc:1035] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.083s
I20260807 11:10:15.963208 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:15.975252 25808 raft_consensus.cc:2379] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 3: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:15.975744 27996 raft_consensus.cc:2726] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:15.964054 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:15.964941 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:15.947775 28088 raft_consensus.cc:2781] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:15.942864 28051 consensus_queue.cc:1046] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260807 11:10:15.939155 27960 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:15.999037 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:16.008388 25919 raft_consensus.cc:1226] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:15.999289 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:15.944425 27963 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:15.983103 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:15.976394 27996 consensus_queue.cc:1035] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 11:10:16.001003 25869 leader_election.cc:302] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:10:16.015995 28033 consensus_meta.cc:287] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.508s user 0.001s sys 0.007s
W20260807 11:10:15.941361 27821 ts_tablet_manager.cc:1462] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543: Trace:
0807 11:10:08.332167 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:08.332457 (+ 290us) service_pool.cc:227] Handling call
0807 11:10:08.602037 (+269580us) ts_tablet_manager.cc:928] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543: Initiating tablet copy from peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195)
0807 11:10:10.104374 (+1502337us) tablet_copy_client.cc:361] Tablet copy session begun
0807 11:10:10.846471 (+742097us) tablet_metadata.cc:641] Metadata flushed
0807 11:10:10.846705 (+ 234us) tablet_copy_client.cc:490] Wrote new tablet metadata
0807 11:10:11.104551 (+257846us) tablet_copy_client.cc:411] Wrote new consensus metadata
0807 11:10:11.164856 (+ 60305us) tablet_replica.cc:205] Creating consensus instance
0807 11:10:11.165346 (+ 490us) ts_tablet_manager.cc:983] Replica registered.
0807 11:10:11.165448 (+ 102us) inbound_call.cc:173] Queueing success response
0807 11:10:12.982508 (+1817060us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0807 11:10:15.283627 (+2301119us) ts_tablet_manager.cc:1429] Starting tablet replica
0807 11:10:15.497733 (+214106us) tablet_replica.cc:259] Starting instrumentation
0807 11:10:15.661415 (+163682us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":492,"raft.queue_time_us":3667,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":6637,"tablet-copy.queue_time_us":264179,"tablet-download-pool.queue_time_us":169,"tablet-download-pool.run_cpu_time_us":2024,"tablet-download-pool.run_wall_time_us":1068340,"thread_start_us":350679,"threads_started":2}
I20260807 11:10:15.908614 28078 raft_consensus.cc:504] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:16.018440 28030 consensus_meta.cc:287] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.738s user 0.006s sys 0.006s
I20260807 11:10:16.019055 25803 raft_consensus.cc:1226] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:16.019763 28078 leader_election.cc:288] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:15.968237 26974 raft_consensus.cc:2454] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:10:16.021703 28039 raft_consensus.cc:3032] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:16.022143 28039 raft_consensus.cc:729] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:16.023108 28039 consensus_queue.cc:260] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:16.024469 28039 raft_consensus.cc:3037] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
W20260807 11:10:16.031200 27817 consensus_meta.cc:287] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.989s user 0.013s sys 0.018s
W20260807 11:10:16.026850 28065 consensus_meta.cc:287] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.526s user 0.006s sys 0.005s
I20260807 11:10:16.031106 25750 leader_election.cc:302] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:16.031881 28065 raft_consensus.cc:504] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:16.007964 27712 consensus_meta.cc:287] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.873s user 0.012s sys 0.009s
I20260807 11:10:15.977073 28025 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:16.029695 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.016429 28033 raft_consensus.cc:504] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:16.031809 27817 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:16.044358 25869 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:16.045630 25800 raft_consensus.cc:1226] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:15.978682 27882 raft_consensus.cc:3037] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:16.028988 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:16.037819 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 73553.1us
I20260807 11:10:16.048895 26041 raft_consensus.cc:2454] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:15.952589 28068 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:16.044409 26037 raft_consensus.cc:2454] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:16.045148 25981 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:16.054183 25981 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:16.056912 25981 leader_election.cc:302] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:16.058663 26986 raft_consensus.cc:1226] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:16.060075 25981 leader_election.cc:302] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:16.063856 25800 raft_consensus.cc:1261] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260807 11:10:16.076319 28073 consensus_meta.cc:287] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.961s user 0.009s sys 0.018s
I20260807 11:10:16.076742 28073 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:16.078745 28073 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:16.079263 27876 raft_consensus.cc:2726] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:16.080101 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.080267 25825 tablet_copy_service.cc:342] P 606d49e053a346e486f48cb31832b815: Request end of tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-66d10b6b12404f0488b9f83c8080069f received from {username='slave'} at 127.0.0.1:59118
I20260807 11:10:16.080839 25825 tablet_copy_service.cc:433] P 606d49e053a346e486f48cb31832b815: ending tablet copy session eb1fca62dcb34ed9b32188ccb19f0543-66d10b6b12404f0488b9f83c8080069f on tablet 66d10b6b12404f0488b9f83c8080069f with peer eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:16.046465 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 128203us
I20260807 11:10:16.060073 28051 raft_consensus.cc:2726] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:16.068272 28056 raft_consensus.cc:2726] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:16.082909 28058 raft_consensus.cc:3032] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:16.083293 28058 raft_consensus.cc:729] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:16.065855 27967 leader_election.cc:288] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:16.047019 28078 raft_consensus.cc:2726] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:16.080788 26037 raft_consensus.cc:2454] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:16.083323 27927 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:16.080744 27960 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:16.085464 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:16.086325 25981 leader_election.cc:302] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:16.087771 25796 raft_consensus.cc:1261] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:16.081255 26978 raft_consensus.cc:2379] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 2: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:16.089263 25798 raft_consensus.cc:2379] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 3: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:16.090911 26938 leader_election.cc:302] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:16.091566 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:16.060559 28030 raft_consensus.cc:2781] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:16.064644 28068 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:16.139031 28065 leader_election.cc:288] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:16.139221 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 48377.1us
I20260807 11:10:16.139698 28080 raft_consensus.cc:482] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:16.140620 28033 leader_election.cc:288] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:16.140300 28080 raft_consensus.cc:504] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:16.141217 28033 raft_consensus.cc:2781] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:16.100298 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:16.141577 28033 raft_consensus.cc:482] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:16.101246 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.141942 28033 raft_consensus.cc:3037] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:16.142220 26041 raft_consensus.cc:2454] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:16.142607 28080 leader_election.cc:288] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:16.143201 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.144541 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.101526 26976 raft_consensus.cc:1261] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:16.101966 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.102246 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:16.102999 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.148092 26038 raft_consensus.cc:2454] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:16.103063 28102 raft_consensus.cc:504] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:16.103847 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.149889 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:16.106024 28067 consensus_meta.cc:287] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.807s user 0.009s sys 0.002s
W20260807 11:10:16.084380 28079 consensus_meta.cc:287] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.611s user 0.003s sys 0.007s
I20260807 11:10:16.106017 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.111199 28092 raft_consensus.cc:504] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:16.111979 28059 raft_consensus.cc:2726] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:16.112409 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.113296 25981 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:16.114208 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.166159 26039 raft_consensus.cc:2454] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:16.166356 26042 raft_consensus.cc:2454] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
W20260807 11:10:16.166605 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 53389.4us
I20260807 11:10:16.167351 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.167680 25981 leader_election.cc:302] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:16.098739 28072 consensus_queue.cc:1035] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 11:10:16.113440 25869 leader_election.cc:302] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:16.104947 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:16.150882 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.044234 27712 raft_consensus.cc:504] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:16.148404 27915 raft_consensus.cc:2726] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:16.154749 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:16.047214 28025 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:16.113153 27733 raft_consensus.cc:2781] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:16.159017 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:16.159161 25917 raft_consensus.cc:1261] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260807 11:10:16.160161 25918 raft_consensus.cc:1226] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:16.160287 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:16.162098 25919 raft_consensus.cc:1261] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:16.169145 26036 raft_consensus.cc:1261] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:16.103201 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:16.063236 28088 raft_consensus.cc:686] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:16.169545 25981 leader_election.cc:302] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:16.170151 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.197937 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:16.198508 26041 raft_consensus.cc:2454] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:16.198803 25919 raft_consensus.cc:2379] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 2: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
W20260807 11:10:16.171660 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 58287.2us
I20260807 11:10:16.198926 25981 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:16.199900 25926 raft_consensus.cc:2379] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 2: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:16.197948 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.145942 28030 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:16.173635 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:16.150274 26046 raft_consensus.cc:2454] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:16.150131 27990 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:16.172127 26035 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:16.206459 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:16.203326 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.174346 28051 raft_consensus.cc:2726] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:16.207379 26037 raft_consensus.cc:2454] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:16.204753 25920 raft_consensus.cc:1226] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:16.178563 27927 raft_consensus.cc:2781] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:16.205885 25869 leader_election.cc:302] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:16.176262 27903 raft_consensus.cc:2726] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:16.198962 27733 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:16.157382 28067 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:16.167472 28079 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:16.198091 28065 consensus_queue.cc:1035] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.085s
I20260807 11:10:16.174968 28119 consensus_queue.cc:1035] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.126s
I20260807 11:10:16.297643 25786 raft_consensus.cc:1226] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:16.214112 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:16.258517 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:16.302309 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:16.309253 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:16.242606 25752 leader_election.cc:334] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.578s (SENT)
I20260807 11:10:16.217613 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:16.302954 26037 raft_consensus.cc:1261] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:16.176707 28058 consensus_queue.cc:260] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:16.316357 26035 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:16.302290 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.258085 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:16.213513 26036 raft_consensus.cc:2454] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:16.334924 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:16.377771 25919 raft_consensus.cc:2379] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 3: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:16.322001 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:16.208359 27876 raft_consensus.cc:2726] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 11:10:16.407068 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 208196us
I20260807 11:10:16.206929 28030 raft_consensus.cc:3037] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:16.402621 26038 raft_consensus.cc:2454] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:16.316838 25917 raft_consensus.cc:3037] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:16.283565 27905 raft_consensus.cc:2726] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 11:10:16.316510 25752 leader_election.cc:302] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:16.404359 28116 consensus_queue.cc:1035] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20260807 11:10:16.290794 28094 consensus_queue.cc:1035] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.142s
I20260807 11:10:16.431308 27907 raft_consensus.cc:504] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:16.431968 28025 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:16.432735 28058 raft_consensus.cc:3037] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:16.220592 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:16.433491 27907 leader_election.cc:288] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:16.401964 25918 raft_consensus.cc:2379] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 3: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:16.402920 25750 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:16.430414 26042 raft_consensus.cc:1261] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:16.242029 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:16.439126 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:16.407222 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:16.407925 25981 leader_election.cc:302] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:16.409526 26987 raft_consensus.cc:1226] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:16.403903 25920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1321) took 1222 ms. Trace:
I20260807 11:10:16.220566 28059 raft_consensus.cc:2726] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:16.211887 27927 raft_consensus.cc:482] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:16.446782 27927 raft_consensus.cc:3037] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:16.416538 28033 raft_consensus.cc:504] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:16.448738 28033 leader_election.cc:288] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:16.449023 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:16.450100 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:16.450706 25918 raft_consensus.cc:3037] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:16.411513 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:10:16.296648 25798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1324) took 1323 ms (client timeout 1720 ms). Trace:
I20260807 11:10:16.410387 25797 raft_consensus.cc:1226] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:16.417207 27817 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:16.417623 28092 leader_election.cc:288] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:16.417796 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.494068 26035 raft_consensus.cc:2454] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:16.418951 28088 consensus_queue.cc:237] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:16.420408 28102 leader_election.cc:288] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:16.308122 27963 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:16.421150 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:16.297294 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:16.340191 27733 raft_consensus.cc:3037] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:16.404738 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.405509 25932 raft_consensus.cc:2454] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:16.295672 28075 consensus_queue.cc:1046] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.127s
I20260807 11:10:16.208372 25931 raft_consensus.cc:1226] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:16.440193 25926 raft_consensus.cc:2379] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 3: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:16.507005 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:16.441439 27712 leader_election.cc:288] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:16.441876 26980 raft_consensus.cc:2379] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 2: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:16.441962 26986 raft_consensus.cc:1203] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 11:10:16.212854 27990 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:16.510887 28096 raft_consensus.cc:3032] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:16.511317 28096 raft_consensus.cc:729] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:16.511631 27990 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:16.511740 25925 raft_consensus.cc:1261] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:16.444722 28079 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:16.438213 28104 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:16.445770 28067 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:16.446030 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:16.499909 28039 raft_consensus.cc:2726] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:16.471587 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:16.472445 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:16.454067 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 300998us
I20260807 11:10:16.477711 25919 raft_consensus.cc:3037] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:16.479542 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:16.460942 26936 leader_election.cc:302] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:16.444646 28055 consensus_meta.cc:287] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.745s user 0.008s sys 0.000s
W20260807 11:10:16.484916 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 243868us
I20260807 11:10:16.478744 26046 raft_consensus.cc:2454] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:16.493021 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
W20260807 11:10:16.493671 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 85835.6us
I20260807 11:10:16.535598 25981 leader_election.cc:302] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:16.499531 25750 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:16.500345 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 275268us
I20260807 11:10:16.503137 26938 leader_election.cc:302] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:16.506029 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:16.440416 25920 rpcz_store.cc:276] 0807 11:10:15.181240 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:15.181500 (+ 260us) service_pool.cc:227] Handling call
0807 11:10:15.186702 (+ 5202us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:16.210468 (+1023766us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:16.403572 (+193104us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b540033a9d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:16.403816 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2331474176}
I20260807 11:10:16.508533 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:16.512374 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:16.493652 28078 raft_consensus.cc:2726] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:16.492213 28025 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:16.556978 28025 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:16.557932 25752 leader_election.cc:334] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.610s (SENT)
W20260807 11:10:16.559051 25752 leader_election.cc:334] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.638s (SENT)
I20260807 11:10:16.559257 28025 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:16.559404 25752 leader_election.cc:302] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:16.484902 25798 rpcz_store.cc:269] 0807 11:10:14.973567 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:14.974429 (+ 862us) service_pool.cc:227] Handling call
0807 11:10:16.296390 (+1321961us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b5400136d54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:16.296562 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2467330432}
I20260807 11:10:16.497148 27882 raft_consensus.cc:504] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:16.552210 27990 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:16.552366 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:16.566089 25926 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:16.552574 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:16.552728 28056 raft_consensus.cc:2726] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:16.553535 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:10:16.554315 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 54882.7us
I20260807 11:10:16.535060 26979 raft_consensus.cc:2379] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 2: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:16.567857 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:16.536504 26038 raft_consensus.cc:2454] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:16.564949 26034 raft_consensus.cc:2454] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:16.561334 25981 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:16.569031 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.569658 25981 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:16.569797 26038 raft_consensus.cc:2454] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
W20260807 11:10:16.562502 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 59522.6us
I20260807 11:10:16.570674 26938 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:16.562757 25917 raft_consensus.cc:2454] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:16.563735 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:16.563776 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:16.554772 28079 raft_consensus.cc:2781] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:16.561101 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:16.562366 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 101497us
I20260807 11:10:16.573021 25932 raft_consensus.cc:2379] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 2: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:16.573207 26042 raft_consensus.cc:2454] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
W20260807 11:10:16.575287 25752 leader_election.cc:334] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.649s (SENT)
I20260807 11:10:16.575770 25752 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:16.585863 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:16.577240 25981 leader_election.cc:302] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:16.647389 25916 raft_consensus.cc:1226] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
W20260807 11:10:16.648378 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 70802us
W20260807 11:10:16.649507 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 72344us
I20260807 11:10:16.650580 25917 raft_consensus.cc:2379] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 2: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:16.650245 25981 leader_election.cc:302] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:16.650287 25932 raft_consensus.cc:1226] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:16.530855 27927 raft_consensus.cc:504] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:16.568044 28067 raft_consensus.cc:2726] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:16.720033 25918 raft_consensus.cc:2454] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:16.574610 28079 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:16.653584 25981 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:16.567657 28114 raft_consensus.cc:2726] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:16.652462 28137 raft_consensus.cc:2726] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:16.605611 28039 raft_consensus.cc:2726] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:16.653874 25869 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:16.681016 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 105751us
W20260807 11:10:16.724843 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 71345.2us
I20260807 11:10:16.576689 28033 raft_consensus.cc:2726] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:16.649202 28030 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:16.604979 28080 raft_consensus.cc:2726] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:16.580027 28059 raft_consensus.cc:2726] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:16.698261 25919 raft_consensus.cc:2454] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:16.722388 25926 raft_consensus.cc:2454] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
W20260807 11:10:16.730942 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 50108.8us
I20260807 11:10:16.605928 28094 consensus_queue.cc:1035] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.113s
I20260807 11:10:16.646555 28078 raft_consensus.cc:2726] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 11:10:16.726168 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 72447.6us
W20260807 11:10:16.726261 25752 leader_election.cc:334] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.626s (SENT)
I20260807 11:10:16.726948 25981 leader_election.cc:302] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:16.728897 25932 raft_consensus.cc:1226] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:16.646121 28055 consensus_queue.cc:1046] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.118s
I20260807 11:10:16.656615 28056 raft_consensus.cc:2726] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:16.733285 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.747290 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.747635 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:16.748013 26035 raft_consensus.cc:2454] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:16.734489 28143 raft_consensus.cc:2726] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:16.748940 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:16.750057 26033 raft_consensus.cc:2454] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:16.751004 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:16.751338 27882 leader_election.cc:288] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:16.751838 26975 raft_consensus.cc:2379] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 2: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:16.744716 28137 raft_consensus.cc:2726] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:16.753438 28075 consensus_queue.cc:1035] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 11:10:16.753991 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.754720 28055 raft_consensus.cc:3032] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:16.755074 28055 raft_consensus.cc:729] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:16.755110 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.755724 26041 raft_consensus.cc:2454] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:16.756233 25932 raft_consensus.cc:1226] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:16.747320 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:16.580874 27990 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:16.763381 25981 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:16.766901 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:16.766963 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:16.738396 25752 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:16.727558 28079 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:16.728071 27876 raft_consensus.cc:2726] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:16.766985 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:16.768860 25806 raft_consensus.cc:2379] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 2: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:16.778949 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.769187 25981 leader_election.cc:302] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:16.770390 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.779793 26038 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:16.772145 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.735455 25750 leader_election.cc:302] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:16.773228 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.780639 25981 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:16.769443 28058 raft_consensus.cc:2726] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:16.809739 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.768252 28114 consensus_queue.cc:1046] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260807 11:10:16.810760 28059 raft_consensus.cc:2781] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:16.811160 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.811906 28068 raft_consensus.cc:3032] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:16.812247 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.813661 25981 leader_election.cc:302] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:16.815351 25869 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:16.818266 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 82911.6us
I20260807 11:10:16.818737 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.819590 26042 raft_consensus.cc:2454] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:16.821666 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.822466 25981 leader_election.cc:302] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:16.822525 26038 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
W20260807 11:10:16.828742 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 102513us
W20260807 11:10:16.829638 25752 leader_election.cc:334] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.630s (SENT)
I20260807 11:10:16.830092 25752 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:16.820698 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.832054 26042 raft_consensus.cc:2454] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:16.834720 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.837266 25981 leader_election.cc:302] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:16.837755 26035 raft_consensus.cc:2454] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:16.839213 25750 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:16.841617 25750 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:16.843402 25752 leader_election.cc:334] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.669s (SENT)
I20260807 11:10:16.844668 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.817405 26978 raft_consensus.cc:2379] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 2: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:16.755684 27733 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:16.852994 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 203888us
I20260807 11:10:16.853751 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.770061 28056 consensus_queue.cc:1046] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.072s
I20260807 11:10:16.855258 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.812319 28068 raft_consensus.cc:729] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:16.811215 28059 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:16.820525 28137 raft_consensus.cc:2726] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:16.861649 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.831272 28065 raft_consensus.cc:2726] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 11:10:16.838806 27905 raft_consensus.cc:2726] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 11:10:16.822009 28152 raft_consensus.cc:2726] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:16.834142 28058 raft_consensus.cc:2726] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:16.874217 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.755935 28055 consensus_queue.cc:260] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:16.835515 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:16.902355 26035 raft_consensus.cc:2454] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:16.847867 27960 raft_consensus.cc:2726] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:16.896118 28059 raft_consensus.cc:3037] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:16.850271 28039 raft_consensus.cc:2726] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:16.851929 28080 raft_consensus.cc:2781] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:16.880167 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:16.854815 27876 raft_consensus.cc:2781] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260807 11:10:16.884806 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 235815us
I20260807 11:10:16.885133 26038 raft_consensus.cc:2454] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:16.890126 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.918362 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
W20260807 11:10:16.900177 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 58671.2us
I20260807 11:10:16.910233 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:16.871592 25752 leader_election.cc:334] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.677s (SENT)
I20260807 11:10:16.913861 26938 leader_election.cc:302] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:16.920677 25752 leader_election.cc:302] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:16.925479 28055 raft_consensus.cc:3037] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:16.917879 27876 raft_consensus.cc:482] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:16.929728 27876 raft_consensus.cc:3037] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:16.930637 28079 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:16.902571 28078 raft_consensus.cc:2781] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:16.932716 28079 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:16.933323 28147 log.cc:924] Time spent T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.100s user 0.000s sys 0.001s
I20260807 11:10:16.933852 28104 raft_consensus.cc:2726] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:16.936023 27733 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:16.941962 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 103397us
I20260807 11:10:16.942838 25750 leader_election.cc:302] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:16.944242 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.944375 26936 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:16.949901 26938 leader_election.cc:302] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:16.951066 26938 leader_election.cc:302] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:16.951782 28039 consensus_queue.cc:1046] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:10:16.952881 28039 raft_consensus.cc:3032] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Stepping down as leader of term 1
W20260807 11:10:16.954704 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 83155.5us
W20260807 11:10:16.956355 25752 leader_election.cc:334] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.691s (SENT)
I20260807 11:10:16.936551 28030 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:16.960532 25981 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:16.963157 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:16.960525 26936 leader_election.cc:302] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:16.947916 28079 raft_consensus.cc:2726] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:16.969821 26936 leader_election.cc:302] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:16.972955 25752 leader_election.cc:334] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.729s (SENT)
I20260807 11:10:16.974126 25752 leader_election.cc:302] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:16.946358 28080 raft_consensus.cc:686] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 2 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:16.978058 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:16.991979 28080 consensus_queue.cc:237] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:16.994287 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:16.996568 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:16.978269 28072 raft_consensus.cc:2726] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:16.978448 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:16.979349 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:17.006805 26041 raft_consensus.cc:2454] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:16.984062 28158 raft_consensus.cc:2781] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:16.955291 27990 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:16.985481 28162 consensus_queue.cc:1046] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 11:10:16.986137 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 1395) took 1067 ms. Trace:
I20260807 11:10:17.013779 26034 rpcz_store.cc:276] 0807 11:10:15.927128 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:15.929714 (+ 2586us) service_pool.cc:227] Handling call
0807 11:10:16.985948 (+1056234us) spinlock_profiling.cc:243] Waited 620 ms on lock 0x7b540038ed54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:16.986049 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1425254144}
I20260807 11:10:16.998109 27960 raft_consensus.cc:2781] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:16.958214 28068 consensus_queue.cc:260] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:17.015992 27960 raft_consensus.cc:686] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 2 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:16.954631 27996 raft_consensus.cc:2726] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:17.017266 27960 consensus_queue.cc:237] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:16.988055 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:16.975589 28030 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:16.990754 26035 raft_consensus.cc:1261] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:16.997395 26986 raft_consensus.cc:3037] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:17.023571 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:16.956192 28150 raft_consensus.cc:2726] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:17.029243 28158 raft_consensus.cc:686] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
W20260807 11:10:17.030272 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 52166.9us
W20260807 11:10:17.033766 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 60863.4us
I20260807 11:10:16.977061 25869 leader_election.cc:302] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:17.002024 28056 raft_consensus.cc:2726] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:17.002478 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:17.035597 26039 raft_consensus.cc:3037] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:16.962316 28067 raft_consensus.cc:2781] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:16.956053 28039 raft_consensus.cc:729] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:16.972193 28157 raft_consensus.cc:2726] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:17.027096 28162 raft_consensus.cc:3032] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:17.006520 27927 leader_election.cc:288] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:17.041224 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 64280us
W20260807 11:10:17.042673 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 82195.5us
I20260807 11:10:17.098346 25800 raft_consensus.cc:1226] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:17.044596 25806 raft_consensus.cc:1226] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:17.099689 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:17.099985 25869 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:17.100553 25919 raft_consensus.cc:1226] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:17.101742 25869 leader_election.cc:302] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:17.011487 28079 raft_consensus.cc:2726] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 11:10:17.103523 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 57813.2us
I20260807 11:10:17.113370 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:17.114063 26041 raft_consensus.cc:2454] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:17.116053 25869 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:17.043644 26974 raft_consensus.cc:1226] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:17.042675 27712 raft_consensus.cc:2726] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:17.113035 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:17.097884 28067 raft_consensus.cc:686] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:17.046842 28162 raft_consensus.cc:729] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:17.130754 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:17.053332 28163 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:17.131954 26985 raft_consensus.cc:2379] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 2: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:17.046446 28078 raft_consensus.cc:686] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:17.043077 28030 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:17.142205 25919 raft_consensus.cc:1226] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:17.109565 27990 raft_consensus.cc:2726] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:17.140213 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:17.042469 27996 consensus_queue.cc:1046] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.082s
I20260807 11:10:17.127115 26975 raft_consensus.cc:2379] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 2: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:17.113520 27927 raft_consensus.cc:2726] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:17.098398 28114 consensus_queue.cc:1035] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260807 11:10:17.027114 28096 consensus_queue.cc:260] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:17.153481 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:17.129777 28125 raft_consensus.cc:2726] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:17.122340 28056 consensus_queue.cc:1046] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20260807 11:10:17.127583 28033 consensus_queue.cc:1046] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.066s
I20260807 11:10:17.203296 28160 consensus_queue.cc:1046] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.077s
I20260807 11:10:17.208580 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:17.146229 28163 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:17.216691 28163 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:17.218626 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:17.219610 26035 raft_consensus.cc:2379] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 2: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:17.220391 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:17.154278 28175 consensus_queue.cc:1046] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.083s
I20260807 11:10:17.228103 28175 raft_consensus.cc:3032] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:17.229574 28178 consensus_queue.cc:1046] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.103s
I20260807 11:10:17.231307 28178 raft_consensus.cc:3032] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:17.231729 28178 raft_consensus.cc:729] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
W20260807 11:10:17.245289 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 210874us
I20260807 11:10:17.228636 28175 raft_consensus.cc:729] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
W20260807 11:10:17.246171 25752 leader_election.cc:334] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.796s (SENT)
I20260807 11:10:17.249009 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:17.255163 26041 raft_consensus.cc:2454] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:17.255249 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:17.255924 25752 leader_election.cc:302] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:17.275477 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:17.246575 28067 consensus_queue.cc:237] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:17.281798 25752 leader_election.cc:334] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.803s (SENT)
I20260807 11:10:17.282346 25752 leader_election.cc:302] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:17.284865 28183 raft_consensus.cc:2726] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:17.247365 28061 raft_consensus.cc:3032] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:17.285738 28061 raft_consensus.cc:729] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:17.281148 28078 consensus_queue.cc:237] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:17.286742 28061 consensus_queue.cc:260] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:17.287958 28061 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:17.290671 25917 raft_consensus.cc:1261] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:17.291210 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1360) took 1126 ms. Trace:
I20260807 11:10:17.296901 25917 rpcz_store.cc:276] 0807 11:10:16.164972 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:16.656833 (+491861us) service_pool.cc:227] Handling call
0807 11:10:17.290237 (+633404us) spinlock_profiling.cc:243] Waited 1.18 ms on lock 0x7b58002e0e98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc461a2b0 00007f3fc46057f9 00007f3fc46052a2 00007f3fbdceb684
0807 11:10:17.290395 (+ 158us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:17.290995 (+ 600us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:17.291165 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2720128}
I20260807 11:10:17.298290 26039 raft_consensus.cc:2454] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
W20260807 11:10:17.282431 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 339651us
I20260807 11:10:17.134711 28158 consensus_queue.cc:237] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:17.297137 28128 log.cc:924] Time spent T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.057s user 0.001s sys 0.000s
I20260807 11:10:17.305568 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:17.304472 27876 raft_consensus.cc:504] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:17.311079 28189 consensus_queue.cc:1035] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.079s
I20260807 11:10:17.312456 28175 consensus_queue.cc:260] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:17.315032 25752 leader_election.cc:334] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.861s (SENT)
I20260807 11:10:17.325651 25752 leader_election.cc:302] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:17.318005 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:17.305263 28163 raft_consensus.cc:2726] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:17.347239 26038 raft_consensus.cc:2379] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 2: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:17.345072 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:17.344871 28030 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:17.348945 25752 leader_election.cc:334] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.901s (SENT)
I20260807 11:10:17.342578 26986 raft_consensus.cc:2454] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:17.410080 25752 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:17.348429 25798 raft_consensus.cc:2379] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 2: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:17.414350 28198 raft_consensus.cc:2726] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 11:10:17.415599 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 66681.5us
W20260807 11:10:17.394256 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 46612.9us
I20260807 11:10:17.414732 28175 raft_consensus.cc:3037] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
W20260807 11:10:17.467382 28149 log.cc:924] Time spent T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.090s user 0.002s sys 0.000s
I20260807 11:10:17.467576 25869 leader_election.cc:302] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:10:17.472064 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 46797us
W20260807 11:10:17.472946 25752 leader_election.cc:334] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.783s (SENT)
W20260807 11:10:17.474150 25752 leader_election.cc:334] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.856s (SENT)
I20260807 11:10:17.474634 25752 leader_election.cc:302] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:17.475503 25872 leader_election.cc:334] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.522s (SENT)
I20260807 11:10:17.474175 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:17.477114 26039 raft_consensus.cc:2379] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 2: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:17.475976 25872 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:17.484180 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 267091us
I20260807 11:10:17.326727 28152 consensus_queue.cc:1046] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.151s
I20260807 11:10:17.468914 28104 raft_consensus.cc:2726] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:17.499593 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:17.500231 26039 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
W20260807 11:10:17.507758 25872 leader_election.cc:334] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.534s (SENT)
I20260807 11:10:17.326084 28178 consensus_queue.cc:260] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:17.488503 27975 raft_consensus.cc:2726] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:17.489929 28030 raft_consensus.cc:2726] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:17.477988 27733 raft_consensus.cc:2781] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:17.525318 25981 leader_election.cc:302] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:17.537358 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:17.538312 26039 raft_consensus.cc:2454] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
W20260807 11:10:17.546176 28177 log.cc:924] Time spent T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.191s user 0.001s sys 0.000s
W20260807 11:10:17.546671 28059 consensus_meta.cc:287] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.617s user 0.010s sys 0.002s
W20260807 11:10:17.549567 28055 consensus_meta.cc:287] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.622s user 0.011s sys 0.000s
I20260807 11:10:17.554577 28075 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
W20260807 11:10:17.564420 28186 log.cc:924] Time spent T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.153s user 0.000s sys 0.000s
I20260807 11:10:17.583550 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:17.565650 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:17.565970 28028 raft_consensus.cc:1050] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:10:17.569100 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:17.569773 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:17.584105 26033 raft_consensus.cc:2454] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:17.570073 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:17.589342 25872 leader_election.cc:334] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.681s (SENT)
I20260807 11:10:17.539588 27733 raft_consensus.cc:686] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:17.597747 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:17.558564 27876 leader_election.cc:288] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:17.630782 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:17.630821 26984 raft_consensus.cc:1226] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:17.580227 28056 consensus_queue.cc:1046] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260807 11:10:17.580256 28200 raft_consensus.cc:2781] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:17.594271 28075 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:17.638094 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 105638us
I20260807 11:10:17.640815 25981 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:17.646935 27733 consensus_queue.cc:237] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:17.581980 28059 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
W20260807 11:10:17.638146 26980 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50960 (request call id 280) took 2032 ms (client timeout 2106 ms). Trace:
W20260807 11:10:17.654831 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 178835us
I20260807 11:10:17.671152 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:17.673269 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 83989.4us
I20260807 11:10:17.666831 28058 raft_consensus.cc:3032] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:17.678573 28058 raft_consensus.cc:729] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:17.665616 25797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53324 (request call id 1312) took 2197 ms. Trace:
I20260807 11:10:17.679117 25797 rpcz_store.cc:276] 0807 11:10:15.468169 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:15.473398 (+ 5229us) service_pool.cc:227] Handling call
0807 11:10:15.492146 (+ 18748us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:16.490635 (+998489us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:17.665204 (+1174569us) spinlock_profiling.cc:243] Waited 918 ms on lock 0x7b5400136854. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:17.665480 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2111216512}
I20260807 11:10:17.679554 28058 consensus_queue.cc:260] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:17.683725 25872 leader_election.cc:334] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.552s (SENT)
I20260807 11:10:17.686388 25872 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:17.688417 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 214291us
W20260807 11:10:17.688607 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 440897us
I20260807 11:10:17.681339 28219 consensus_queue.cc:1046] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260807 11:10:17.691444 25752 leader_election.cc:334] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.907s (SENT)
I20260807 11:10:17.710955 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:17.714751 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:17.662220 28056 raft_consensus.cc:3037] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
W20260807 11:10:17.714989 25872 leader_election.cc:334] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.616s (SENT)
I20260807 11:10:17.717561 25917 raft_consensus.cc:1226] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:17.713423 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:17.671180 26974 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 322) took 1194 ms. Trace:
I20260807 11:10:17.646430 28200 raft_consensus.cc:482] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:17.721825 26033 raft_consensus.cc:2454] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:17.739024 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:17.734052 25919 raft_consensus.cc:1226] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:17.736021 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:17.666215 28075 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:17.744346 25926 raft_consensus.cc:1261] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:10:17.712248 28080 consensus_queue.cc:1035] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260807 11:10:17.748507 28059 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:17.749390 28059 consensus_queue.cc:1046] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:17.699594 28068 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:17.741518 26039 raft_consensus.cc:2454] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
W20260807 11:10:17.762956 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 71530.7us
W20260807 11:10:17.767040 25752 leader_election.cc:334] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.913s (SENT)
I20260807 11:10:17.762648 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:17.714320 27712 raft_consensus.cc:2726] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 11:10:17.779668 25752 leader_election.cc:334] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.970s (SENT)
I20260807 11:10:17.783664 25752 leader_election.cc:302] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:17.781845 25918 raft_consensus.cc:1226] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:17.715852 25931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 93) took 1140 ms. Trace:
I20260807 11:10:17.793167 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:17.799357 25801 raft_consensus.cc:1226] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:17.764390 28059 consensus_queue.cc:1046] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260807 11:10:17.738277 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:17.670058 28201 consensus_queue.cc:1046] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.190s
W20260807 11:10:17.674800 26980 rpcz_store.cc:269] 0807 11:10:15.604051 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:15.605637 (+ 1586us) service_pool.cc:227] Handling call
0807 11:10:17.630516 (+2024879us) spinlock_profiling.cc:243] Waited 744 ms on lock 0x7b54004718d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:17.635914 (+ 5398us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1711813888}
I20260807 11:10:17.743791 28200 raft_consensus.cc:3037] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:17.849001 26037 raft_consensus.cc:1261] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:17.798135 28114 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:17.851049 25801 raft_consensus.cc:1226] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
W20260807 11:10:17.854981 25872 leader_election.cc:334] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.708s (SENT)
W20260807 11:10:17.855383 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 75757.3us
I20260807 11:10:17.855515 25872 leader_election.cc:302] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:17.838883 28080 raft_consensus.cc:2726] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260807 11:10:17.866567 25872 leader_election.cc:334] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.721s (SENT)
I20260807 11:10:17.867174 25872 leader_election.cc:302] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:17.735001 26974 rpcz_store.cc:276] 0807 11:10:16.477010 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:16.478592 (+ 1582us) service_pool.cc:227] Handling call
0807 11:10:17.017386 (+538794us) spinlock_profiling.cc:243] Waited 4.72 ms on lock 0x7b5800000f98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc461a2b0 00007f3fc46057f9 00007f3fc46052a2 00007f3fbdceb684
0807 11:10:17.017661 (+ 275us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:17.127696 (+110035us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:17.670844 (+543148us) spinlock_profiling.cc:243] Waited 4.92 ms on lock 0x7b54002c1354. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:17.671059 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22167680}
W20260807 11:10:17.876250 25872 leader_election.cc:334] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.763s (SENT)
I20260807 11:10:17.876950 25872 leader_election.cc:302] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:17.873268 28088 raft_consensus.cc:2726] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:17.842628 28152 consensus_queue.cc:1035] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260807 11:10:17.865140 28059 raft_consensus.cc:3032] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:17.893905 26034 raft_consensus.cc:2454] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:17.900911 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:17.844691 28067 consensus_queue.cc:1046] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260807 11:10:17.908622 26033 raft_consensus.cc:2379] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 2: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:17.868472 28061 consensus_queue.cc:1035] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
W20260807 11:10:17.930564 25752 leader_election.cc:334] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.921s (SENT)
I20260807 11:10:17.896375 28059 raft_consensus.cc:729] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:17.899066 28088 raft_consensus.cc:2726] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:17.862643 28114 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:17.940356 25872 leader_election.cc:334] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.799s (SENT)
I20260807 11:10:17.941083 25872 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:17.949275 28211 log.cc:924] Time spent T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20260807 11:10:17.949676 28122 raft_consensus.cc:2726] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:17.933252 25752 leader_election.cc:302] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:17.956128 26985 raft_consensus.cc:1226] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:17.973119 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:17.974999 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 44473us
W20260807 11:10:17.975870 25752 leader_election.cc:334] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.987s (SENT)
W20260807 11:10:17.987850 25752 leader_election.cc:334] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 2.108s (SENT)
I20260807 11:10:17.988829 25752 leader_election.cc:302] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:17.988920 25872 leader_election.cc:334] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.906s (SENT)
I20260807 11:10:17.933226 28102 consensus_queue.cc:1046] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260807 11:10:17.955717 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
W20260807 11:10:18.014046 25872 leader_election.cc:334] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 2.167s (SENT)
I20260807 11:10:18.038599 25872 leader_election.cc:302] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:18.042682 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:18.043622 26038 raft_consensus.cc:2454] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:18.047694 26984 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 320) took 1609 ms. Trace:
I20260807 11:10:18.051056 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:18.051992 26038 raft_consensus.cc:2362] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2. Re-sending same reply.
I20260807 11:10:18.053176 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:18.054631 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:17.797086 25931 rpcz_store.cc:276] 0807 11:10:16.575706 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:16.833332 (+257626us) service_pool.cc:227] Handling call
0807 11:10:16.876838 (+ 43506us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:16.892533 (+ 15695us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:16.894625 (+ 2092us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:16.931351 (+ 36726us) log.cc:842] Serialized 26 byte log entry
0807 11:10:16.964660 (+ 33309us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:17.121312 (+156652us) log.cc:842] Serialized 14 byte log entry
0807 11:10:17.150133 (+ 28821us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:17.150232 (+ 99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:17.682856 (+532624us) raft_consensus.cc:1671] finished
0807 11:10:17.682881 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:17.711458 (+ 28577us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15120,"prepare.queue_time_us":174884,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":35858,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":5328,"thread_start_us":148483,"threads_started":2,"wal-append.queue_time_us":109005}
I20260807 11:10:18.057538 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:18.048300 26984 rpcz_store.cc:276] 0807 11:10:16.438138 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:16.438265 (+ 127us) service_pool.cc:227] Handling call
0807 11:10:17.630638 (+1192373us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:17.634233 (+ 3595us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:18.047450 (+413217us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54004718d0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:18.047607 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2734717824}
W20260807 11:10:18.057607 28212 log.cc:924] Time spent T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.135s user 0.000s sys 0.001s
W20260807 11:10:18.070307 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 83369.8us
W20260807 11:10:18.071208 25752 leader_election.cc:334] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 2.148s (SENT)
I20260807 11:10:18.071820 25752 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:18.082629 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 68649.9us
W20260807 11:10:18.092871 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 51112.8us
I20260807 11:10:18.097278 28080 raft_consensus.cc:1050] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:10:18.070111 28061 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:18.079546 27963 raft_consensus.cc:2726] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 11:10:18.108721 25752 leader_election.cc:334] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 2.205s (SENT)
W20260807 11:10:18.112700 25752 leader_election.cc:334] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 2.225s (SENT)
I20260807 11:10:18.113132 25752 leader_election.cc:302] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:18.113329 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:18.103343 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:18.113969 26038 raft_consensus.cc:2454] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
W20260807 11:10:18.117347 25872 leader_election.cc:334] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.969s (SENT)
I20260807 11:10:18.117902 25872 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:18.037910 28183 consensus_queue.cc:1046] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
W20260807 11:10:18.126013 25872 leader_election.cc:334] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 2.012s (SENT)
W20260807 11:10:18.126752 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 430425us
I20260807 11:10:18.126950 28080 consensus_queue.cc:237] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:18.115787 28152 raft_consensus.cc:2726] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:18.166507 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:18.167793 28229 raft_consensus.cc:2726] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 11:10:18.169359 28217 log.cc:924] Time spent T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.134s user 0.000s sys 0.000s
I20260807 11:10:18.153553 28092 raft_consensus.cc:2726] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:18.179776 28114 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:18.169371 28216 log.cc:924] Time spent T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.277s user 0.000s sys 0.002s
I20260807 11:10:18.189018 27975 raft_consensus.cc:2726] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:18.189808 28163 raft_consensus.cc:3032] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:18.194175 28137 raft_consensus.cc:482] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:18.197054 28178 raft_consensus.cc:3037] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:18.199064 28096 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:18.085444 28059 consensus_queue.cc:260] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:18.116323 28061 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:18.211599 28106 raft_consensus.cc:1050] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:10:18.213500 27817 raft_consensus.cc:1050] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:10:18.215649 25925 raft_consensus.cc:1226] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
W20260807 11:10:18.204816 28175 consensus_meta.cc:287] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.738s user 0.015s sys 0.000s
I20260807 11:10:18.219856 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:18.181946 28206 raft_consensus.cc:2726] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:18.178790 28203 consensus_queue.cc:1046] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.282s
I20260807 11:10:18.236848 28039 consensus_queue.cc:260] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:18.238993 28059 raft_consensus.cc:3037] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:18.239992 28190 raft_consensus.cc:3032] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:18.241087 28079 raft_consensus.cc:3032] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:18.206663 28163 raft_consensus.cc:729] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:18.242597 27876 raft_consensus.cc:3032] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:18.279918 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:10:18.285633 28219 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.057s sys 0.098s
W20260807 11:10:18.286428 28219 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.547s user 0.058s sys 0.098s
I20260807 11:10:18.232227 28137 raft_consensus.cc:504] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:18.294930 28137 leader_election.cc:288] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:18.317963 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:18.325686 26033 raft_consensus.cc:2379] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 2: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:18.266577 28079 raft_consensus.cc:729] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
W20260807 11:10:18.280953 27733 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.618s user 0.032s sys 0.048s
I20260807 11:10:18.335829 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59312 (request call id 1301) took 1314 ms. Trace:
I20260807 11:10:18.290079 28190 raft_consensus.cc:729] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:18.336352 26034 rpcz_store.cc:276] 0807 11:10:17.021841 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:17.022036 (+ 195us) service_pool.cc:227] Handling call
0807 11:10:18.335614 (+1313578us) spinlock_profiling.cc:243] Waited 558 ms on lock 0x7b54000ba754. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:18.335756 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1283516160}
W20260807 11:10:18.314836 28068 consensus_meta.cc:287] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.536s user 0.009s sys 0.005s
I20260807 11:10:18.278815 27876 raft_consensus.cc:729] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:18.350064 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:18.297610 28039 raft_consensus.cc:3037] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
W20260807 11:10:18.329263 27733 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.670s user 0.033s sys 0.049s
I20260807 11:10:18.268719 28163 consensus_queue.cc:260] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:18.385638 25872 leader_election.cc:334] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.837s (SENT)
I20260807 11:10:18.388221 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:10:18.393759 28200 consensus_meta.cc:287] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.538s user 0.000s sys 0.014s
I20260807 11:10:18.395612 27876 consensus_queue.cc:260] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:18.395800 28200 raft_consensus.cc:504] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:18.397487 27876 raft_consensus.cc:3037] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:18.414994 28190 consensus_queue.cc:260] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:18.418308 28068 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:18.438315 28068 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:18.411912 28079 consensus_queue.cc:260] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:18.449136 26938 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:18.451022 28033 raft_consensus.cc:2781] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:18.451764 28033 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:18.452314 28033 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:18.458994 28219 raft_consensus.cc:482] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:18.408634 28088 raft_consensus.cc:2726] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:16.994330 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:10:18.480684 26036 raft_consensus.cc:2454] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:18.490280 28200 leader_election.cc:288] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:18.491117 28200 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:18.491883 28200 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:18.495515 28200 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:18.496248 28200 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:18.496974 28200 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:18.500111 28200 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:18.501286 28221 raft_consensus.cc:482] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:18.501941 28221 raft_consensus.cc:504] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:18.502532 27817 consensus_queue.cc:237] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:18.511471 25658 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34692 (request call id 133) took 7440 ms. Trace:
I20260807 11:10:18.446853 28163 raft_consensus.cc:3037] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
W20260807 11:10:18.531428 26037 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.600s user 0.005s sys 0.022s
I20260807 11:10:18.486521 28079 raft_consensus.cc:3037] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:18.480633 28206 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:18.480125 28175 raft_consensus.cc:482] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:18.486162 28061 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195)
W20260807 11:10:18.537436 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 386192us
I20260807 11:10:18.512001 25658 rpcz_store.cc:276] 0807 11:10:11.071255 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:11.071384 (+ 129us) service_pool.cc:227] Handling call
0807 11:10:18.511390 (+7440006us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 11:10:16.083868 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 11:10:16.089729 (+ 5861us) write_op.cc:432] Acquiring schema lock in shared mode
0807 11:10:16.089776 (+ 47us) write_op.cc:435] Acquired schema lock
0807 11:10:16.089796 (+ 20us) tablet.cc:660] Decoding operations
0807 11:10:16.090375 (+ 579us) write_op.cc:620] Acquiring the partition lock for write op
0807 11:10:16.090459 (+ 84us) write_op.cc:641] Partition lock acquired for write op
0807 11:10:16.090485 (+ 26us) tablet.cc:683] Acquiring locks for 3 operations
0807 11:10:16.090813 (+ 328us) tablet.cc:699] Row locks acquired
0807 11:10:16.090832 (+ 19us) write_op.cc:260] PREPARE: finished
0807 11:10:16.090985 (+ 153us) write_op.cc:270] Start()
0807 11:10:16.091098 (+ 113us) write_op.cc:276] Timestamp: P: 1786101016090944 usec, L: 0
0807 11:10:16.091123 (+ 25us) op_driver.cc:331] REPLICATION: starting
0807 11:10:16.098972 (+ 7849us) log.cc:842] Serialized 1311 byte log entry
0807 11:10:16.446119 (+347147us) op_driver.cc:444] REPLICATION: finished
0807 11:10:16.564525 (+118406us) write_op.cc:301] APPLY: starting
0807 11:10:16.567320 (+ 2795us) tablet.cc:1365] starting BulkCheckPresence
0807 11:10:16.650466 (+ 83146us) tablet.cc:1368] finished BulkCheckPresence
0807 11:10:16.723675 (+ 73209us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 11:10:16.809501 (+ 85826us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 11:10:16.850887 (+ 41386us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0807 11:10:16.862075 (+ 11188us) write_op.cc:312] APPLY: finished
0807 11:10:16.893545 (+ 31470us) log.cc:842] Serialized 35 byte log entry
0807 11:10:16.899958 (+ 6413us) write_op.cc:489] Releasing partition, row and schema locks
0807 11:10:16.911449 (+ 11491us) write_op.cc:454] Released schema lock
0807 11:10:16.912921 (+ 1472us) write_op.cc:341] FINISH: Updating metrics
0807 11:10:16.928913 (+ 15992us) write_op.cc:489] Releasing partition, row and schema locks
0807 11:10:16.928955 (+ 42us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2670,"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":118078,"apply.run_cpu_time_us":3761,"apply.run_wall_time_us":364725,"num_ops":3,"prepare.queue_time_us":118329,"prepare.run_cpu_time_us":3527,"prepare.run_wall_time_us":42964,"replication_time_us":354868,"thread_start_us":118116,"threads_started":2}]]}
I20260807 11:10:18.509392 28190 raft_consensus.cc:3037] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:18.504691 28221 leader_election.cc:288] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:10:18.532078 26037 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.601s user 0.005s sys 0.022s
I20260807 11:10:18.513713 28203 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:18.545459 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 432754us
W20260807 11:10:18.532051 28116 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.006s sys 0.031s
W20260807 11:10:18.559545 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 422044us
I20260807 11:10:18.567214 25916 raft_consensus.cc:1261] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:10:18.567356 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:18.568176 26038 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 1.
W20260807 11:10:18.557641 28116 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.572s user 0.006s sys 0.032s
I20260807 11:10:18.583771 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 94) took 1092 ms. Trace:
I20260807 11:10:18.584152 26046 rpcz_store.cc:276] 0807 11:10:17.485625 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:17.486058 (+ 433us) service_pool.cc:227] Handling call
0807 11:10:17.499872 (+ 13814us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:17.500175 (+ 303us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:17.500200 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:17.500595 (+ 395us) log.cc:842] Serialized 26 byte log entry
0807 11:10:17.712539 (+211944us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:18.109670 (+397131us) log.cc:842] Serialized 14 byte log entry
0807 11:10:18.577838 (+468168us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:18.577920 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:18.577967 (+ 47us) raft_consensus.cc:1671] finished
0807 11:10:18.577976 (+ 9us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:18.578094 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":640586,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":122263,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":285,"thread_start_us":1017701,"threads_started":3,"wal-append.queue_time_us":347120}
I20260807 11:10:18.544997 28068 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:18.597362 25919 raft_consensus.cc:1226] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:18.538084 28175 raft_consensus.cc:504] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:18.569259 28200 raft_consensus.cc:482] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:18.521920 28219 raft_consensus.cc:504] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:18.613060 25800 raft_consensus.cc:1226] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:18.608106 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:18.573124 28227 raft_consensus.cc:482] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:18.615475 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:18.612300 25981 leader_election.cc:302] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:18.618146 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:18.538355 28206 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:18.622179 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:18.623536 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:18.627740 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:18.581403 28152 consensus_queue.cc:1046] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260807 11:10:18.648244 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 96) took 1095 ms. Trace:
I20260807 11:10:18.548731 28203 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:18.648923 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:18.610874 28068 raft_consensus.cc:482] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:18.627280 28056 consensus_meta.cc:287] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.893s user 0.010s sys 0.005s
I20260807 11:10:18.635845 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:18.629225 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:18.625093 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:18.648659 26041 rpcz_store.cc:276] 0807 11:10:17.553195 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:17.553385 (+ 190us) service_pool.cc:227] Handling call
0807 11:10:17.553794 (+ 409us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:17.912059 (+358265us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:17.912101 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:17.923861 (+ 11760us) log.cc:842] Serialized 26 byte log entry
0807 11:10:18.355245 (+431384us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:18.390005 (+ 34760us) log.cc:842] Serialized 14 byte log entry
0807 11:10:18.390412 (+ 407us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:18.390514 (+ 102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:18.648044 (+257530us) raft_consensus.cc:1671] finished
0807 11:10:18.648063 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:18.648188 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5159,"prepare.queue_time_us":34581,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":72891,"raft.run_cpu_time_us":372,"raft.run_wall_time_us":10289,"thread_start_us":758490,"threads_started":2,"wal-append.queue_time_us":613176}
I20260807 11:10:18.665071 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:18.583421 28237 consensus_queue.cc:1046] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.067s
W20260807 11:10:18.669754 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 168115us
I20260807 11:10:18.649616 28220 raft_consensus.cc:3032] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:18.677872 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:18.680076 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:18.687474 25801 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.613s user 0.016s sys 0.029s
W20260807 11:10:18.687832 25801 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.613s user 0.016s sys 0.029s
I20260807 11:10:18.615660 28200 raft_consensus.cc:504] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:18.689596 26038 raft_consensus.cc:2454] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:18.666205 26976 raft_consensus.cc:1226] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:18.692651 25920 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:18.693377 26041 raft_consensus.cc:1261] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:10:18.692883 26046 raft_consensus.cc:1261] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:10:18.619442 28227 raft_consensus.cc:504] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:18.697404 28150 raft_consensus.cc:482] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:18.673161 28056 raft_consensus.cc:2726] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:18.705888 28206 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:18.708086 28175 leader_election.cc:288] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:18.693266 28220 raft_consensus.cc:729] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:18.719558 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:18.720461 28258 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:18.650390 27963 consensus_queue.cc:1035] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260807 11:10:18.736784 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:18.737525 25919 raft_consensus.cc:2454] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:18.739177 25750 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:18.660061 28068 raft_consensus.cc:504] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:18.666257 28033 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:18.689983 28257 consensus_queue.cc:1046] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.073s
W20260807 11:10:18.732169 28096 consensus_meta.cc:287] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.500s user 0.006s sys 0.006s
I20260807 11:10:18.735824 28175 raft_consensus.cc:2726] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:18.764607 26984 raft_consensus.cc:1226] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:18.764622 25872 leader_election.cc:302] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:18.765988 27882 raft_consensus.cc:2781] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:18.766440 27882 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:18.766862 27882 raft_consensus.cc:3037] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:18.780956 26039 raft_consensus.cc:2454] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:18.760381 28073 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:18.782680 28073 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:18.790019 25872 leader_election.cc:302] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:18.794093 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:18.755415 28065 raft_consensus.cc:2781] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260807 11:10:18.808283 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 228484us
I20260807 11:10:18.819001 25872 leader_election.cc:302] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:18.730165 28150 raft_consensus.cc:504] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:18.795835 28231 log.cc:924] Time spent T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.192s user 0.002s sys 0.000s
I20260807 11:10:18.791020 28157 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:18.790040 28219 leader_election.cc:288] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:18.734973 28258 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:18.796777 28068 leader_election.cc:288] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:18.859278 26042 raft_consensus.cc:2454] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:18.841634 25872 leader_election.cc:302] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:18.797628 28065 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:18.797580 28122 raft_consensus.cc:2726] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:18.851449 28247 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:18.828177 27963 raft_consensus.cc:482] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:18.849263 28257 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:18.907629 28205 raft_consensus.cc:3032] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:18.908103 28205 raft_consensus.cc:729] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:18.908718 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:18.910493 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:18.883216 28065 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:18.879704 28200 leader_election.cc:288] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:18.957221 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:18.958494 25925 raft_consensus.cc:2454] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:10:18.960767 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:18.961725 25801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1399) took 1007 ms. Trace:
I20260807 11:10:18.962133 25801 rpcz_store.cc:276] 0807 11:10:17.953948 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:17.955692 (+ 1744us) service_pool.cc:227] Handling call
0807 11:10:17.992535 (+ 36843us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:17.998334 (+ 5799us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:17.998386 (+ 52us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:18.038183 (+ 39797us) log.cc:842] Serialized 26 byte log entry
0807 11:10:18.692525 (+654342us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:10:18.692562 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:18.693831 (+ 1269us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:18.961499 (+267668us) raft_consensus.cc:1671] finished
0807 11:10:18.961523 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:18.961653 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":613452,"threads_started":1,"wal-append.queue_time_us":858258}
W20260807 11:10:18.964519 28265 log.cc:924] Time spent T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20260807 11:10:18.856364 28028 consensus_queue.cc:237] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:18.897964 28227 leader_election.cc:288] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:18.978375 25918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1388) took 1175 ms. Trace:
I20260807 11:10:18.904660 28122 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:18.983469 25918 rpcz_store.cc:276] 0807 11:10:17.803299 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:17.838066 (+ 34767us) service_pool.cc:227] Handling call
0807 11:10:17.854027 (+ 15961us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:17.863409 (+ 9382us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:17.863451 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:17.875603 (+ 12152us) log.cc:842] Serialized 572 byte log entry
0807 11:10:18.402268 (+526665us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:18.402320 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:18.402422 (+ 102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:18.978085 (+575663us) raft_consensus.cc:1671] finished
0807 11:10:18.978120 (+ 35us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:18.978285 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":479264,"threads_started":1,"wal-append.queue_time_us":661096}
I20260807 11:10:18.858208 28157 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:19.000414 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:18.880648 28263 raft_consensus.cc:2726] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:19.014791 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:19.019850 25926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 98) took 1305 ms. Trace:
I20260807 11:10:19.022724 25920 raft_consensus.cc:2454] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:19.025290 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:19.030740 25925 raft_consensus.cc:1261] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:19.034132 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:19.020498 25926 rpcz_store.cc:276] 0807 11:10:17.714319 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:17.745345 (+ 31026us) service_pool.cc:227] Handling call
0807 11:10:18.461141 (+715796us) spinlock_profiling.cc:243] Waited 1.2 ms on lock 0x7b58002e0e98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc461a2b0 00007f3fc46057f9 00007f3fc46052a2 00007f3fbdceb684
0807 11:10:18.461433 (+ 292us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:18.461795 (+ 362us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:18.461838 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:18.485252 (+ 23414us) log.cc:842] Serialized 26 byte log entry
0807 11:10:18.754220 (+268968us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:18.799509 (+ 45289us) log.cc:842] Serialized 14 byte log entry
0807 11:10:18.799696 (+ 187us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:18.833262 (+ 33566us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:19.019369 (+186107us) raft_consensus.cc:1671] finished
0807 11:10:19.019569 (+ 200us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:19.019752 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":45109,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"raft.queue_time_us":19391,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":21332,"spinlock_wait_cycles":72423552,"thread_start_us":222226,"threads_started":1,"wal-append.queue_time_us":467747}
I20260807 11:10:19.009460 28122 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:18.901028 27963 raft_consensus.cc:504] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:19.035127 26046 raft_consensus.cc:2454] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:19.047390 26938 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:18.986526 28203 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:18.908963 28205 consensus_queue.cc:260] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:19.066772 26036 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59312 (request call id 1326) took 1498 ms (client timeout 1901 ms). Trace:
W20260807 11:10:19.067274 26036 rpcz_store.cc:269] 0807 11:10:17.568158 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:17.710605 (+142447us) service_pool.cc:227] Handling call
0807 11:10:19.066603 (+1355998us) spinlock_profiling.cc:243] Waited 534 ms on lock 0x7b54002df854. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:19.066709 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1228882048}
I20260807 11:10:18.918672 28257 raft_consensus.cc:504] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:19.075347 28160 raft_consensus.cc:2781] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:19.055213 28033 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:19.056095 28252 raft_consensus.cc:2726] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 11:10:19.083292 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 471039us
I20260807 11:10:19.088021 28150 leader_election.cc:288] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:19.062800 28258 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:19.062867 28122 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:19.146492 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:19.079558 28160 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:19.180231 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:19.176739 28252 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:19.183302 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:18.899977 28247 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:19.125707 26980 consensus_meta.cc:287] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 1.047s user 0.001s sys 0.015s
I20260807 11:10:19.092682 28205 raft_consensus.cc:3037] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:19.190462 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:19.119961 28244 raft_consensus.cc:482] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:19.081261 27882 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:19.181118 28160 raft_consensus.cc:3037] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:19.194579 25981 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:19.123860 27733 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:19.202036 28247 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:19.205313 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:19.245456 25916 raft_consensus.cc:3037] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:19.206393 28065 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:19.208715 28220 consensus_queue.cc:260] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:19.214512 28183 raft_consensus.cc:3032] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Stepping down as leader of term 1
W20260807 11:10:19.223551 26042 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59312 (request call id 1320) took 1658 ms (client timeout 1500 ms). Trace:
I20260807 11:10:19.186206 28252 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:19.251318 26042 rpcz_store.cc:269] 0807 11:10:17.565507 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:17.568838 (+ 3331us) service_pool.cc:227] Handling call
0807 11:10:19.223304 (+1654466us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b540009f454. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:19.223467 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2619351808}
I20260807 11:10:19.254130 28183 raft_consensus.cc:729] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
W20260807 11:10:19.200594 28190 consensus_meta.cc:287] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.657s user 0.013s sys 0.001s
W20260807 11:10:19.238139 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 43656.9us
I20260807 11:10:19.242097 28192 raft_consensus.cc:2781] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:19.195706 26036 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a"
dest_uuid: "b22d69db45194908ad3d345eba9badd9"
mode: GRACEFUL
new_leader_uuid: "4fd6381544574ddca559ce63b9243efc"
from {username='slave'} at 127.0.0.1:58132
I20260807 11:10:19.260567 26036 raft_consensus.cc:595] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Received request to transfer leadership to TS 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:19.249063 28258 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:19.233923 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:19.268199 27882 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:19.269155 28192 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:19.269820 28192 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:19.263003 26038 raft_consensus.cc:2454] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:19.280839 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:19.261678 28163 consensus_meta.cc:287] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.728s user 0.008s sys 0.006s
W20260807 11:10:19.263361 27876 consensus_meta.cc:287] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.865s user 0.006s sys 0.008s
I20260807 11:10:19.262669 28119 raft_consensus.cc:482] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:19.294237 25800 raft_consensus.cc:2454] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:19.255326 28183 consensus_queue.cc:260] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:19.245785 27733 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:19.319939 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:19.271807 28157 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:19.340240 28257 leader_election.cc:288] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:10:19.342046 28274 log.cc:924] Time spent T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.186s user 0.000s sys 0.001s
I20260807 11:10:19.271394 28258 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:19.346434 28258 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:19.349411 28252 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:19.350275 28073 leader_election.cc:288] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:19.350680 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:19.314870 28119 raft_consensus.cc:504] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:19.332412 28039 consensus_meta.cc:287] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.940s user 0.011s sys 0.003s
I20260807 11:10:19.354259 28065 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:19.356698 26976 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 329) took 1407 ms. Trace:
W20260807 11:10:19.357227 28079 consensus_meta.cc:287] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.822s user 0.007s sys 0.007s
I20260807 11:10:19.357362 26976 rpcz_store.cc:276] 0807 11:10:17.949293 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:17.955051 (+ 5758us) service_pool.cc:227] Handling call
0807 11:10:18.212108 (+257057us) spinlock_profiling.cc:243] Waited 3.24 ms on lock 0x7b5800000f98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc461a2b0 00007f3fc46057f9 00007f3fc46052a2 00007f3fbdceb684
0807 11:10:18.212408 (+ 300us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:18.712958 (+500550us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:19.356278 (+643320us) spinlock_profiling.cc:243] Waited 437 ms on lock 0x7b54006b44d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:19.356570 (+ 292us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1012644608}
I20260807 11:10:19.361655 28058 raft_consensus.cc:3037] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:19.312302 28246 raft_consensus.cc:2932] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:19.300134 25916 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:19.266076 28244 raft_consensus.cc:504] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:19.277463 27963 leader_election.cc:288] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:19.311601 28183 raft_consensus.cc:3037] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:19.364784 26035 raft_consensus.cc:2454] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:19.235960 26980 raft_consensus.cc:2454] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:19.368692 28162 consensus_queue.cc:260] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:19.372835 28162 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
W20260807 11:10:19.375487 26980 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 325) took 1462 ms (client timeout 1737 ms). Trace:
I20260807 11:10:19.377857 25917 raft_consensus.cc:1226] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:19.373930 28221 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:19.380971 26035 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59312 (request call id 1313) took 2097 ms (client timeout 1906 ms). Trace:
W20260807 11:10:19.382593 26035 rpcz_store.cc:269] 0807 11:10:17.283880 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:17.284006 (+ 126us) service_pool.cc:227] Handling call
0807 11:10:19.380761 (+2096755us) spinlock_profiling.cc:243] Waited 2.05 s on lock 0x7b54000a3554. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:19.380906 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4711954176}
W20260807 11:10:19.377188 26980 rpcz_store.cc:269] 0807 11:10:17.913270 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:17.921444 (+ 8174us) service_pool.cc:227] Handling call
0807 11:10:19.374307 (+1452863us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:19.383428 28244 leader_election.cc:288] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:19.380198 28221 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:19.405158 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:19.405736 27733 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:19.416481 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 154062us
W20260807 11:10:19.422516 26936 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.020s sys 0.058s
W20260807 11:10:19.422864 26936 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.509s user 0.020s sys 0.058s
I20260807 11:10:19.437944 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:10:19.421612 26037 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.879s user 0.094s sys 0.188s
I20260807 11:10:19.441716 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:19.445281 26976 raft_consensus.cc:2379] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 2: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:19.451279 25799 raft_consensus.cc:2454] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
W20260807 11:10:19.440435 26037 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.899s user 0.094s sys 0.189s
W20260807 11:10:19.440153 27817 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.791s user 0.020s sys 0.063s
W20260807 11:10:19.455859 27817 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.810s user 0.021s sys 0.063s
I20260807 11:10:19.462188 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:19.463203 25916 raft_consensus.cc:2454] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:19.473331 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:19.475176 25920 raft_consensus.cc:1226] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:19.474282 25917 raft_consensus.cc:2454] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:19.485519 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:19.486522 25917 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:10:19.496083 27817 raft_consensus.cc:482] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:19.497469 28220 raft_consensus.cc:3037] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:19.497215 27817 raft_consensus.cc:504] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:19.454946 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:19.512034 28119 leader_election.cc:288] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:19.518028 25919 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.022s sys 0.047s
I20260807 11:10:19.512378 27817 leader_election.cc:288] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:19.518497 25919 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.552s user 0.022s sys 0.047s
W20260807 11:10:19.520915 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 102687us
I20260807 11:10:19.525413 26936 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:10:19.531534 25872 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.641s user 0.025s sys 0.019s
W20260807 11:10:19.532016 25872 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.646s user 0.026s sys 0.019s
W20260807 11:10:19.508059 28156 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.854s user 0.019s sys 0.041s
I20260807 11:10:19.533547 28239 raft_consensus.cc:482] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:19.524505 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 101) took 1649 ms. Trace:
I20260807 11:10:19.534335 28239 raft_consensus.cc:504] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:19.532019 25981 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:10:19.537925 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 696331us
I20260807 11:10:19.539263 25872 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:19.508535 28221 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:10:19.536922 25869 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.651s user 0.046s sys 0.071s
I20260807 11:10:19.434795 28192 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:19.540338 28237 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.815s user 0.018s sys 0.053s
I20260807 11:10:19.536667 28114 raft_consensus.cc:2781] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:19.525333 28055 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:19.546957 26936 leader_election.cc:302] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:10:19.546988 25932 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.957s user 0.034s sys 0.069s
I20260807 11:10:19.550886 26936 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:19.544579 25872 leader_election.cc:302] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
W20260807 11:10:19.543210 25869 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.657s user 0.046s sys 0.071s
W20260807 11:10:19.540191 28156 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.935s user 0.020s sys 0.041s
W20260807 11:10:19.551800 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 706118us
I20260807 11:10:19.535210 26037 rpcz_store.cc:276] 0807 11:10:17.875234 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:17.875467 (+ 233us) service_pool.cc:227] Handling call
0807 11:10:17.899770 (+ 24303us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:17.900056 (+ 286us) raft_consensus.cc:1520] Early marking committed up to index 5
0807 11:10:18.541268 (+641212us) log.cc:842] Serialized 14 byte log entry
0807 11:10:19.521624 (+980356us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:19.521699 (+ 75us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:19.521761 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:19.524234 (+ 2473us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:19.524414 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":453276,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":27,"raft.queue_time_us":66279,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":11392,"thread_start_us":1522883,"threads_started":2}
I20260807 11:10:19.547238 26980 raft_consensus.cc:2379] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 2: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:19.556088 26936 leader_election.cc:302] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:10:19.551033 25932 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.961s user 0.035s sys 0.069s
I20260807 11:10:19.561739 28215 raft_consensus.cc:2781] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:19.561939 28125 raft_consensus.cc:2781] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:19.562281 28215 raft_consensus.cc:482] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:19.562434 28125 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:19.562915 28215 raft_consensus.cc:3037] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:19.563093 28125 raft_consensus.cc:3037] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:19.561199 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 97) took 1847 ms. Trace:
I20260807 11:10:19.545722 28137 raft_consensus.cc:2781] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:19.546595 27817 raft_consensus.cc:2781] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:19.559542 25981 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:19.568346 25981 leader_election.cc:302] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:19.571135 25917 raft_consensus.cc:1226] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:19.574280 26936 leader_election.cc:302] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:19.575246 25872 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:19.549906 28114 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 11:10:19.551946 28237 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.830s user 0.019s sys 0.053s
I20260807 11:10:19.545876 28061 consensus_queue.cc:1035] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260807 11:10:19.554397 28158 raft_consensus.cc:2781] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:19.565119 25932 rpcz_store.cc:276] 0807 11:10:17.713933 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:17.739721 (+ 25788us) service_pool.cc:227] Handling call
0807 11:10:18.299925 (+560204us) spinlock_profiling.cc:243] Waited 960 us on lock 0x7b58002e0e98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc461a2b0 00007f3fc46057f9 00007f3fc46052a2 00007f3fbdceb684
0807 11:10:18.300123 (+ 198us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:18.300310 (+ 187us) raft_consensus.cc:1520] Early marking committed up to index 6
0807 11:10:18.589488 (+289178us) log.cc:842] Serialized 14 byte log entry
0807 11:10:19.560700 (+971212us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:19.560791 (+ 91us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:10:19.560851 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:19.560942 (+ 91us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:19.561122 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3192,"prepare.queue_time_us":392339,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":4019,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":153,"spinlock_wait_cycles":2208128,"thread_start_us":1228257,"threads_started":2}
W20260807 11:10:19.581238 26937 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.536s user 0.017s sys 0.006s
I20260807 11:10:19.584179 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:19.587953 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:19.586387 26937 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.541s user 0.018s sys 0.006s
I20260807 11:10:19.585448 25872 leader_election.cc:302] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:19.567518 28203 raft_consensus.cc:2781] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:19.592619 25872 leader_election.cc:302] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:19.583582 28059 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:19.572422 27733 raft_consensus.cc:465] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Not starting pre-election -- already a leader
I20260807 11:10:19.581563 28114 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:19.573294 27817 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:19.581254 28190 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:19.579773 28039 raft_consensus.cc:2726] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:19.576910 28221 raft_consensus.cc:2726] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:19.553423 28160 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:19.602814 25872 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:19.598460 25983 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.831s user 0.037s sys 0.067s
I20260807 11:10:19.581349 28137 raft_consensus.cc:686] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 2 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:19.555106 28055 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:19.612079 26938 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.026s sys 0.018s
W20260807 11:10:19.612546 26938 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.560s user 0.026s sys 0.018s
I20260807 11:10:19.584800 28158 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:19.602355 28094 raft_consensus.cc:2781] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:19.592818 28203 raft_consensus.cc:482] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:19.620236 25981 leader_election.cc:302] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:19.595153 27927 raft_consensus.cc:2726] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:19.597544 28252 raft_consensus.cc:2726] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:19.603672 28201 raft_consensus.cc:482] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:19.612993 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 565677us
I20260807 11:10:19.620148 28094 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:19.605665 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:19.623553 28279 log.cc:924] Time spent T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
I20260807 11:10:19.608052 27733 raft_consensus.cc:2726] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:19.632455 26041 raft_consensus.cc:1226] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:19.601999 28275 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
W20260807 11:10:19.622583 25983 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.909s user 0.038s sys 0.067s
I20260807 11:10:19.606041 27817 raft_consensus.cc:3037] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:19.625115 28203 raft_consensus.cc:3037] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:19.597420 28059 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:19.611155 28156 raft_consensus.cc:482] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:19.638999 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:19.622005 28158 raft_consensus.cc:3037] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:19.647675 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:10:19.647972 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 45263.1us
W20260807 11:10:19.650444 25750 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.641s user 0.011s sys 0.026s
W20260807 11:10:19.674803 25750 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.675s user 0.013s sys 0.026s
W20260807 11:10:19.675207 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 675557us
I20260807 11:10:19.675422 26046 raft_consensus.cc:2454] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:19.675608 25872 leader_election.cc:302] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:19.634089 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:19.654304 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:19.676800 26035 raft_consensus.cc:2454] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:19.670445 26034 raft_consensus.cc:2454] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:19.630173 28094 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:19.655045 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:19.680125 26037 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:19.646329 28280 raft_consensus.cc:3037] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:19.678573 26984 raft_consensus.cc:2379] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 3: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:19.678452 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:19.624017 28137 consensus_queue.cc:237] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:19.611320 28190 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
W20260807 11:10:19.673763 25751 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.673s user 0.024s sys 0.024s
W20260807 11:10:19.735637 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 60107.1us
W20260807 11:10:19.763058 25751 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.762s user 0.024s sys 0.024s
I20260807 11:10:19.764866 25872 leader_election.cc:302] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:19.765218 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:19.737120 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 53486.4us
I20260807 11:10:19.737382 25917 raft_consensus.cc:1226] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:19.741201 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:10:19.650195 25752 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.664s user 0.002s sys 0.035s
W20260807 11:10:19.766484 25752 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.780s user 0.003s sys 0.035s
I20260807 11:10:19.770821 25918 raft_consensus.cc:2379] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 2: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:19.770450 25872 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:19.648245 25925 raft_consensus.cc:2454] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:19.772261 25872 leader_election.cc:302] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:19.740015 25875 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.655s user 0.003s sys 0.033s
W20260807 11:10:19.766937 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 807705us
I20260807 11:10:19.654237 28286 raft_consensus.cc:2726] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:19.680222 28175 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:19.643512 28201 raft_consensus.cc:504] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:19.805596 25872 leader_election.cc:302] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:19.806237 25875 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.921s user 0.004s sys 0.033s
I20260807 11:10:19.680475 28178 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:19.685626 28258 raft_consensus.cc:482] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:19.786966 25917 raft_consensus.cc:1261] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260807 11:10:19.791762 26936 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:19.679324 28058 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:19.809628 26938 leader_election.cc:302] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:19.810250 25750 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:19.738262 28079 raft_consensus.cc:482] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:19.818436 25981 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:19.761085 28206 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:19.819625 26936 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:19.758689 28244 raft_consensus.cc:482] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:19.769212 28122 raft_consensus.cc:2781] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:19.735129 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:19.758669 28078 raft_consensus.cc:482] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:19.819105 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:19.904604 25916 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:19.905180 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:19.805297 28088 raft_consensus.cc:2726] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:19.686553 28156 raft_consensus.cc:504] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:19.829562 28072 raft_consensus.cc:2726] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:19.782092 28192 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:19.785126 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
W20260807 11:10:19.842972 25755 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.658s user 0.005s sys 0.029s
I20260807 11:10:19.847826 25872 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:19.815451 27882 raft_consensus.cc:2726] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:19.815150 28065 raft_consensus.cc:482] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:19.905887 25800 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:19.777174 28242 consensus_queue.cc:1046] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.069s
I20260807 11:10:19.822086 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:19.918690 28078 raft_consensus.cc:504] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:19.830947 28025 raft_consensus.cc:2726] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:19.665145 28275 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:19.982710 28055 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:19.821434 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1390) took 1835 ms. Trace:
W20260807 11:10:19.849561 28285 log.cc:924] Time spent T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.176s user 0.000s sys 0.001s
I20260807 11:10:19.828895 28258 raft_consensus.cc:504] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:19.820940 28110 consensus_queue.cc:1046] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.100s
W20260807 11:10:19.964161 28268 log.cc:924] Time spent T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.350s user 0.000s sys 0.001s
I20260807 11:10:19.903868 28206 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:19.843624 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:19.845698 28061 raft_consensus.cc:2781] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:19.836586 28283 raft_consensus.cc:2781] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:20.039343 28283 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:19.843410 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:19.816558 28175 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.045153 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:20.045776 28175 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:20.046459 25786 raft_consensus.cc:1261] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 11:10:19.845573 28293 raft_consensus.cc:2781] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:20.047585 28293 raft_consensus.cc:686] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:20.048744 28293 consensus_queue.cc:237] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:19.976155 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
W20260807 11:10:19.973284 28028 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.609s user 0.059s sys 0.086s
I20260807 11:10:19.977950 28059 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:19.907143 28122 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:19.822882 28178 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:19.924312 26985 raft_consensus.cc:1261] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260807 11:10:19.983883 25755 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.998s user 0.008s sys 0.033s
I20260807 11:10:19.986572 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:20.037627 28061 raft_consensus.cc:482] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:19.997522 25916 raft_consensus.cc:2454] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
W20260807 11:10:19.999836 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 152102us
W20260807 11:10:20.057489 28028 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.693s user 0.060s sys 0.086s
I20260807 11:10:19.838259 28079 raft_consensus.cc:504] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:19.849820 28294 consensus_queue.cc:1046] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.144s
I20260807 11:10:19.915634 28244 raft_consensus.cc:504] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:19.841482 28209 raft_consensus.cc:2726] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:19.987749 28160 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:20.042912 28283 raft_consensus.cc:3037] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:20.007232 27882 raft_consensus.cc:2781] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:20.018237 28102 raft_consensus.cc:3032] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:19.999107 28065 raft_consensus.cc:504] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.039703 28110 raft_consensus.cc:3032] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:19.833375 28058 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.001380 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.144865 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:20.069815 25797 raft_consensus.cc:1261] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:20.071946 28237 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:20.071965 25872 leader_election.cc:302] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:20.065766 28055 raft_consensus.cc:482] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:20.073726 28106 raft_consensus.cc:1054] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc: Unable to promote non-voter eb1fca62dcb34ed9b32188ccb19f0543: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }.
Pending config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:20.075028 28296 consensus_queue.cc:1035] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.226s
I20260807 11:10:20.075496 26978 raft_consensus.cc:1261] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 LEARNER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260807 11:10:20.080917 28278 log.cc:924] Time spent T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.189s user 0.000s sys 0.001s
I20260807 11:10:20.081492 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:20.082769 28300 log.cc:924] Time spent T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
W20260807 11:10:20.083326 28291 log.cc:924] Time spent T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.232s user 0.000s sys 0.001s
I20260807 11:10:20.084722 28258 leader_election.cc:288] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:20.088203 28178 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:20.088781 28030 consensus_queue.cc:1035] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:10:20.197029 25919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1402) took 1396 ms. Trace:
I20260807 11:10:20.021564 25921 rpcz_store.cc:276] 0807 11:10:17.980871 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:18.033097 (+ 52226us) service_pool.cc:227] Handling call
0807 11:10:18.101016 (+ 67919us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:18.601059 (+500043us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:18.603146 (+ 2087us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:18.670544 (+ 67398us) log.cc:842] Serialized 26 byte log entry
0807 11:10:19.092223 (+421679us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:10:19.121038 (+ 28815us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:19.151878 (+ 30840us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:19.778020 (+626142us) raft_consensus.cc:1671] finished
0807 11:10:19.786690 (+ 8670us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:19.816033 (+ 29343us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":295,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":6196,"thread_start_us":869604,"threads_started":2,"wal-append.queue_time_us":808729}
I20260807 11:10:20.197675 28065 leader_election.cc:288] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:20.198351 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:20.092370 28215 consensus_meta.cc:287] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.529s user 0.006s sys 0.002s
I20260807 11:10:20.068022 28122 raft_consensus.cc:3037] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:20.208709 28056 raft_consensus.cc:3032] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:20.209254 28056 raft_consensus.cc:729] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:20.097206 28206 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:20.098287 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:20.040541 28190 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:20.112238 28079 leader_election.cc:288] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:20.113415 28229 raft_consensus.cc:1013] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: attempt to promote peer eb1fca62dcb34ed9b32188ccb19f0543: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260807 11:10:20.113888 28175 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:20.114952 28239 leader_election.cc:288] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:20.196233 28094 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:20.103154 25750 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
W20260807 11:10:20.239557 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 136513us
W20260807 11:10:20.118461 26034 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59312 (request call id 1342) took 1872 ms (client timeout 1963 ms). Trace:
W20260807 11:10:20.242748 26034 rpcz_store.cc:269] 0807 11:10:18.246296 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:18.349735 (+103439us) service_pool.cc:227] Handling call
0807 11:10:20.118286 (+1768551us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54000a4954. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:20.118402 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2872603392}
I20260807 11:10:20.122505 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:20.122568 25981 leader_election.cc:302] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:20.123407 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.123989 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:20.245630 26035 raft_consensus.cc:3037] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
W20260807 11:10:20.245710 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 123220us
I20260807 11:10:20.246687 25981 leader_election.cc:302] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:19.890357 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 60465.4us
I20260807 11:10:20.099809 28061 raft_consensus.cc:3037] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
W20260807 11:10:20.108157 28162 consensus_meta.cc:287] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.733s user 0.005s sys 0.007s
I20260807 11:10:20.100262 28201 leader_election.cc:288] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:10:20.123353 28183 consensus_meta.cc:287] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.751s user 0.011s sys 0.003s
W20260807 11:10:20.123548 28277 log.cc:924] Time spent T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.363s user 0.001s sys 0.001s
I20260807 11:10:20.072134 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:20.120059 28246 consensus_meta.cc:287] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.740s user 0.003s sys 0.009s
I20260807 11:10:20.280164 26986 raft_consensus.cc:2454] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
W20260807 11:10:20.147459 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 75585.7us
I20260807 11:10:20.229424 28198 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:20.229275 28206 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:20.281334 25872 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:20.260193 28239 raft_consensus.cc:2781] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:20.283526 28239 raft_consensus.cc:686] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 2 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:20.108631 28242 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:20.284518 28239 consensus_queue.cc:237] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:20.285120 28242 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.287261 28242 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:20.280143 28201 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:20.153364 25786 raft_consensus.cc:1226] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 4. Ops: []
I20260807 11:10:20.153491 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.154300 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.289737 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:20.290182 25800 raft_consensus.cc:2454] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:20.291045 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.185403 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:10:20.188607 28114 consensus_meta.cc:287] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.578s user 0.010s sys 0.000s
I20260807 11:10:20.142752 27882 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:20.300496 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.301333 25800 raft_consensus.cc:2454] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:20.190243 28058 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:20.197381 25919 rpcz_store.cc:276] 0807 11:10:18.800292 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:18.800575 (+ 283us) service_pool.cc:227] Handling call
0807 11:10:18.836416 (+ 35841us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:18.846350 (+ 9934us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:18.846387 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:18.885361 (+ 38974us) log.cc:842] Serialized 26 byte log entry
0807 11:10:19.526560 (+641199us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:10:19.526599 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:19.526679 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:20.196837 (+670158us) raft_consensus.cc:1671] finished
0807 11:10:20.196857 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:20.196973 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2453,"raft.queue_time_us":49329,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":20918,"thread_start_us":559689,"threads_started":1,"wal-append.queue_time_us":833124}
I20260807 11:10:20.142400 28102 raft_consensus.cc:729] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:20.130069 28294 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:20.144115 28110 raft_consensus.cc:729] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:20.212702 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:20.155978 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:20.226168 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:20.226164 25869 leader_election.cc:302] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:20.227180 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.201115 26985 raft_consensus.cc:2932] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:20.183197 28028 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:20.107268 26037 raft_consensus.cc:2454] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:10:20.153550 28286 raft_consensus.cc:2781] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:20.203537 28078 leader_election.cc:288] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:20.251694 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.213820 28156 leader_election.cc:288] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:20.253415 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.211216 28178 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:20.254321 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:20.346791 25808 raft_consensus.cc:3037] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:20.202768 28296 raft_consensus.cc:482] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:20.260862 26034 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "737af11139a541029b9473a8bfdb5980"
dest_uuid: "b22d69db45194908ad3d345eba9badd9"
mode: GRACEFUL
new_leader_uuid: "606d49e053a346e486f48cb31832b815"
from {username='slave'} at 127.0.0.1:58132
I20260807 11:10:20.349743 26034 raft_consensus.cc:595] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Received request to transfer leadership to TS 606d49e053a346e486f48cb31832b815
I20260807 11:10:20.350102 26034 raft_consensus.cc:603] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 11:10:20.341295 26980 raft_consensus.cc:2379] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 4: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:20.154758 28158 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:20.181229 28055 raft_consensus.cc:504] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.158632 28237 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.229611 28065 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:20.290128 25797 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:10:20.294710 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:20.296200 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:20.254696 28190 raft_consensus.cc:2726] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:20.343122 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:20.288324 28201 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.300869 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:20.290899 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:20.222138 28215 raft_consensus.cc:504] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:20.314815 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.264271 28106 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:20.319186 25803 raft_consensus.cc:2454] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:20.408959 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:20.409673 25803 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:20.236967 28175 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.332906 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:20.319849 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 93758.6us
I20260807 11:10:20.335110 25981 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:20.414505 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 79508.6us
I20260807 11:10:20.338346 26938 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:20.340978 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.345556 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:20.305045 27882 raft_consensus.cc:3037] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:20.291241 28119 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:20.423491 26041 raft_consensus.cc:2454] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:20.347731 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.255825 25750 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:20.426878 25806 raft_consensus.cc:2454] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:20.295879 27733 raft_consensus.cc:2726] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:20.300701 28088 raft_consensus.cc:2781] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:20.300933 28079 raft_consensus.cc:2781] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:20.300146 28257 raft_consensus.cc:482] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:20.363426 25919 raft_consensus.cc:1226] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:20.373867 26039 raft_consensus.cc:3037] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:20.298161 28206 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.374825 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:20.425347 28125 consensus_meta.cc:287] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.862s user 0.008s sys 0.005s
I20260807 11:10:20.261942 28056 consensus_queue.cc:260] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:20.263147 27817 consensus_meta.cc:287] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.590s user 0.009s sys 0.002s
I20260807 11:10:20.299784 28114 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:20.350431 28275 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:20.225248 28160 consensus_queue.cc:1035] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260807 11:10:20.297695 28198 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.357734 28250 raft_consensus.cc:2781] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:20.382761 28065 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:20.400192 28303 log.cc:924] Time spent T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.086s user 0.000s sys 0.001s
I20260807 11:10:20.401587 28237 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:20.351871 28178 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.402935 28055 leader_election.cc:288] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:20.403980 28158 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:20.337435 28294 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.406018 28122 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.491357 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.411402 26936 leader_election.cc:302] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:10:20.493088 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 81768.5us
I20260807 11:10:20.495324 28072 raft_consensus.cc:2781] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:20.495877 28072 raft_consensus.cc:482] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:20.496302 28072 raft_consensus.cc:3037] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:20.407969 25917 raft_consensus.cc:2932] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:20.423233 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.498577 25796 raft_consensus.cc:2454] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:10:20.499468 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.499531 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:20.500130 25796 raft_consensus.cc:2454] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:10:20.500200 25932 raft_consensus.cc:3037] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:20.463927 28125 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.505300 28125 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:20.505960 25926 raft_consensus.cc:1226] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:20.506606 28065 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:20.507035 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.507671 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:20.508199 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.508841 25981 leader_election.cc:302] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
W20260807 11:10:20.424901 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 86692.2us
I20260807 11:10:20.512655 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.427186 26037 raft_consensus.cc:1261] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260807 11:10:20.427290 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 171532us
I20260807 11:10:20.513480 25799 raft_consensus.cc:2454] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:20.426662 26035 raft_consensus.cc:2454] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:10:20.382972 28244 leader_election.cc:288] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:20.344626 28028 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.432569 25786 raft_consensus.cc:1226] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:20.366140 28296 raft_consensus.cc:504] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.519265 26938 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:20.443329 25806 raft_consensus.cc:1226] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:20.340873 28059 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:20.446556 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:20.422353 28058 raft_consensus.cc:2726] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 11:10:20.420594 28286 raft_consensus.cc:686] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:20.461009 25808 raft_consensus.cc:2454] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:10:20.425962 28192 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:20.428681 28078 raft_consensus.cc:2726] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:20.433322 28297 raft_consensus.cc:2781] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:20.366022 28102 consensus_queue.cc:260] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:20.454922 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.466284 25869 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:20.457784 28203 consensus_meta.cc:287] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.778s user 0.008s sys 0.002s
I20260807 11:10:20.417730 28106 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.475255 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.478950 26039 raft_consensus.cc:2454] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:20.451097 28094 leader_election.cc:288] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:20.481264 26987 raft_consensus.cc:1226] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:20.468997 27817 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:20.483662 28319 log.cc:924] Time spent T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.051s user 0.000s sys 0.001s
I20260807 11:10:20.475839 28339 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:20.490863 28294 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:20.514957 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.515352 25981 leader_election.cc:302] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:20.563943 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:20.564659 26037 raft_consensus.cc:3037] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
W20260807 11:10:20.526981 28220 consensus_meta.cc:287] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 1.029s user 0.004s sys 0.010s
I20260807 11:10:20.433542 28119 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:20.413983 26976 raft_consensus.cc:1226] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 3. Ops: []
I20260807 11:10:20.520455 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.522102 28088 raft_consensus.cc:686] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 3 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:20.525241 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:20.569535 26042 raft_consensus.cc:2454] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:20.463984 28079 raft_consensus.cc:686] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:20.493155 28122 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:20.527452 27927 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:20.428251 28110 consensus_queue.cc:260] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:20.505976 28315 log.cc:924] Time spent T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.224s user 0.000s sys 0.001s
I20260807 11:10:20.510705 28061 raft_consensus.cc:504] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:20.581475 28079 consensus_queue.cc:237] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:20.583103 28061 leader_election.cc:288] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:20.453562 28257 raft_consensus.cc:504] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.584892 28028 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:20.585610 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:20.532414 28296 leader_election.cc:288] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:20.587050 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:20.587803 25921 raft_consensus.cc:2454] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
W20260807 11:10:20.528194 28316 log.cc:924] Time spent T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.121s user 0.000s sys 0.002s
I20260807 11:10:20.448256 28025 consensus_queue.cc:1035] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260807 11:10:20.589742 28198 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:20.589953 27817 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:20.569428 28229 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:20.590842 25918 raft_consensus.cc:1226] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:20.592144 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:20.593050 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:20.593659 25796 raft_consensus.cc:3037] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:20.555354 28114 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:20.555764 28215 leader_election.cc:288] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:20.556934 28102 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:20.556743 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:20.528007 25798 raft_consensus.cc:2454] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:20.529649 28192 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.556777 28206 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:20.557435 25931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:39730 (request call id 683) took 2464 ms. Trace:
I20260807 11:10:20.545608 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.569733 28058 raft_consensus.cc:2781] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:20.546540 28283 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.557849 28297 raft_consensus.cc:686] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:20.558215 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:20.542587 28106 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:20.517858 28055 raft_consensus.cc:2726] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:20.562620 25803 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
W20260807 11:10:20.565502 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 99348.4us
I20260807 11:10:20.546784 28203 raft_consensus.cc:504] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:20.479846 28250 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:20.618402 28250 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:20.618644 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:20.570188 25919 raft_consensus.cc:3037] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:20.528280 28059 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.574869 25932 raft_consensus.cc:2454] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:10:20.537999 28158 raft_consensus.cc:2781] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:20.579613 25752 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:20.585789 28257 leader_election.cc:288] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:20.636967 28244 raft_consensus.cc:2726] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:20.637153 28178 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:20.613637 28175 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:20.519425 27960 consensus_queue.cc:1035] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20260807 11:10:20.535245 25806 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f"
dest_uuid: "606d49e053a346e486f48cb31832b815"
mode: GRACEFUL
new_leader_uuid: "4fd6381544574ddca559ce63b9243efc"
from {username='slave'} at 127.0.0.1:58114
I20260807 11:10:20.639153 25806 raft_consensus.cc:595] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Received request to transfer leadership to TS 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:20.597272 26037 raft_consensus.cc:2454] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:10:20.598278 26936 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:20.595207 27927 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:20.564675 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 49402us
I20260807 11:10:20.572316 28072 raft_consensus.cc:504] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:20.640695 25981 leader_election.cc:302] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:20.641697 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:20.642208 25981 leader_election.cc:302] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:20.600569 28201 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:20.626348 25796 raft_consensus.cc:2454] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:10:20.595252 28242 consensus_queue.cc:1046] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260807 11:10:20.628537 25919 raft_consensus.cc:2454] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
W20260807 11:10:20.630609 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 51096.2us
I20260807 11:10:20.634608 26976 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50960 (request call id 291) took 1019 ms. Trace:
I20260807 11:10:20.635803 28059 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:20.605072 28058 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:20.601812 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:20.643008 26936 leader_election.cc:302] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:20.670254 26936 leader_election.cc:302] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:20.645923 28158 raft_consensus.cc:686] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:20.603701 25931 rpcz_store.cc:276] 0807 11:10:18.092612 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:18.104359 (+ 11747us) service_pool.cc:227] Handling call
0807 11:10:20.519871 (+2415512us) spinlock_profiling.cc:243] Waited 1.95 s on lock 0x7b54001df954. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649af0 00007f3fbf6ccf2a 00007f3fc4609c6e 00007f3fbdd00807 00007f3fbdd00792 00007f3fbdd006c1 00007f3fbdd0063c 00007f3fbdcff8f2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:20.557347 (+ 37476us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4482901888}
I20260807 11:10:20.603369 28192 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:20.600407 28294 consensus_queue.cc:1035] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260807 11:10:20.646080 25981 leader_election.cc:302] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:20.685428 25981 leader_election.cc:302] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:20.686478 25981 leader_election.cc:302] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:20.648334 28141 raft_consensus.cc:2726] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 11:10:20.568508 28339 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:20.603638 28206 consensus_queue.cc:1046] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260807 11:10:20.647593 25752 leader_election.cc:302] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:20.647661 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:20.690649 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:20.691634 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:20.665547 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:20.666246 26938 leader_election.cc:302] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:20.647243 28175 raft_consensus.cc:2726] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:20.695014 28311 raft_consensus.cc:2781] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:20.619005 28297 consensus_queue.cc:237] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:20.689364 28178 raft_consensus.cc:2726] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:20.695992 28311 raft_consensus.cc:686] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:20.669611 28058 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:20.673435 28157 raft_consensus.cc:2781] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:20.674885 28203 leader_election.cc:288] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:20.675642 27927 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:20.541625 28286 consensus_queue.cc:237] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.683542 26936 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:20.688905 25981 leader_election.cc:302] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:20.692627 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:20.645449 27882 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.664644 26976 rpcz_store.cc:276] 0807 11:10:19.615268 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:19.622934 (+ 7666us) service_pool.cc:227] Handling call
0807 11:10:19.623531 (+ 597us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:20.570523 (+946992us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:20.625241 (+ 54718us) spinlock_profiling.cc:243] Waited 760 ms on lock 0x7b54005dded4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:20.634539 (+ 9298us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1748092800}
I20260807 11:10:20.647990 28201 raft_consensus.cc:2726] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:20.698642 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:20.715806 26041 raft_consensus.cc:3037] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:20.691816 28055 raft_consensus.cc:2726] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:20.699196 28094 raft_consensus.cc:2781] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader election won for term 3
W20260807 11:10:20.704335 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 56834us
I20260807 11:10:20.718139 28283 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:20.724655 25752 leader_election.cc:302] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:20.714375 28242 raft_consensus.cc:3032] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:20.726505 25752 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:20.727089 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:20.705619 25981 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:20.727648 25752 leader_election.cc:302] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:20.707020 28339 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:20.707144 28141 raft_consensus.cc:2781] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:20.707506 28072 leader_election.cc:288] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:20.710368 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:20.694933 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:20.755457 26980 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 295) took 1141 ms. Trace:
I20260807 11:10:20.756100 26980 rpcz_store.cc:276] 0807 11:10:19.615068 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:19.629332 (+ 14264us) service_pool.cc:227] Handling call
0807 11:10:20.755191 (+1125859us) spinlock_profiling.cc:243] Waited 358 ms on lock 0x7b54004c13d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:20.755379 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":822540032}
I20260807 11:10:20.756727 28094 raft_consensus.cc:686] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:20.711440 28175 raft_consensus.cc:2726] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:20.758204 28094 consensus_queue.cc:237] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:20.714138 28280 consensus_meta.cc:287] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.950s user 0.009s sys 0.004s
I20260807 11:10:20.721160 28157 raft_consensus.cc:686] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:20.714690 28065 raft_consensus.cc:2781] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:20.769272 28065 raft_consensus.cc:482] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:20.769845 28065 raft_consensus.cc:3037] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:20.772882 28280 raft_consensus.cc:2726] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:20.773674 26038 raft_consensus.cc:2454] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:20.772919 28141 raft_consensus.cc:686] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:20.774076 28157 consensus_queue.cc:237] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:20.727632 25806 raft_consensus.cc:3037] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:20.693627 28192 raft_consensus.cc:2726] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:20.728559 25981 leader_election.cc:302] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:20.782949 28198 raft_consensus.cc:3032] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:20.725579 28294 raft_consensus.cc:2726] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:20.725627 28242 raft_consensus.cc:729] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:20.788007 28192 raft_consensus.cc:2781] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:20.788582 28192 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:20.743391 25808 raft_consensus.cc:3032] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:20.789273 25808 raft_consensus.cc:729] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:20.743279 28058 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.666249 28088 consensus_queue.cc:237] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.731983 25752 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:20.748169 28205 raft_consensus.cc:2726] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260807 11:10:20.789379 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 60905us
I20260807 11:10:20.807752 25981 leader_election.cc:302] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:20.750150 28206 raft_consensus.cc:3037] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:20.748840 28283 raft_consensus.cc:2781] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260807 11:10:20.799181 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 67308.9us
I20260807 11:10:20.814705 25981 leader_election.cc:302] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:20.815301 25752 leader_election.cc:302] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:20.807137 25808 consensus_queue.cc:260] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:20.699656 28311 consensus_queue.cc:237] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:20.817062 25808 raft_consensus.cc:3037] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:20.817219 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43246 (request call id 115) took 1183 ms. Trace:
W20260807 11:10:20.820084 26986 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 331) took 2102 ms (client timeout 2129 ms). Trace:
I20260807 11:10:20.796145 28192 raft_consensus.cc:3037] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:20.797250 28280 raft_consensus.cc:482] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:20.796465 28198 raft_consensus.cc:729] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
W20260807 11:10:20.849879 25750 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.120.132:43279 timed out after 2.129s (SENT) [suppressed 74 similar messages]
I20260807 11:10:20.849761 26042 rpcz_store.cc:276] 0807 11:10:19.633635 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:19.633781 (+ 146us) service_pool.cc:227] Handling call
0807 11:10:20.816995 (+1183214us) spinlock_profiling.cc:243] Waited 893 ms on lock 0x7b540038e354. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:20.817138 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2053637888}
W20260807 11:10:20.849844 26986 rpcz_store.cc:269] 0807 11:10:18.714409 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:18.714569 (+ 160us) service_pool.cc:227] Handling call
0807 11:10:20.816908 (+2102339us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b54006edf54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:20.817015 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3577867648}
I20260807 11:10:20.808118 28292 raft_consensus.cc:2726] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:20.854146 25752 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:20.815721 28039 raft_consensus.cc:2781] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:20.818210 28283 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:20.876698 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:20.876936 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:20.877585 26034 raft_consensus.cc:2379] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 2: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:20.818704 28201 raft_consensus.cc:2726] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:20.881755 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:20.820991 28175 consensus_queue.cc:1046] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20260807 11:10:20.767943 28158 consensus_queue.cc:237] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:20.875615 26046 raft_consensus.cc:1226] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 2 LEADER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:20.895593 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:20.818439 28293 raft_consensus.cc:2726] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:20.896836 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:20.897092 26035 raft_consensus.cc:1226] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:20.880498 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:20.867910 25981 leader_election.cc:302] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:20.882529 25752 leader_election.cc:302] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:20.885862 28246 consensus_queue.cc:1035] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:20.886659 28250 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:20.900028 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:20.887163 28110 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:20.888113 25918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1412) took 1111 ms. Trace:
I20260807 11:10:20.874141 28039 raft_consensus.cc:482] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:20.888744 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:20.908113 28039 raft_consensus.cc:3037] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:20.900672 26046 raft_consensus.cc:2454] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:20.908598 25981 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:20.909482 28178 raft_consensus.cc:2726] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:20.909668 28065 raft_consensus.cc:504] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:20.900949 25808 raft_consensus.cc:1261] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 3. (term mismatch)
I20260807 11:10:20.902855 25918 rpcz_store.cc:276] 0807 11:10:19.778066 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:19.780160 (+ 2094us) service_pool.cc:227] Handling call
0807 11:10:19.780462 (+ 302us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:20.591336 (+810874us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:20.887866 (+296530us) spinlock_profiling.cc:243] Waited 810 ms on lock 0x7b5400407754. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:20.888051 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1863143680}
I20260807 11:10:20.911485 25808 pending_rounds.cc:76] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815: Aborting all ops after (but not including) 2
I20260807 11:10:20.911860 25808 pending_rounds.cc:97] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 1.3
I20260807 11:10:20.912352 25808 raft_consensus.cc:2894] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted by new leader
I20260807 11:10:20.914546 25808 raft_consensus.cc:2866] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:10:20.915162 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
W20260807 11:10:20.915530 28163 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260807 11:10:20.915632 26987 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 335) took 1176 ms. Trace:
I20260807 11:10:20.916064 26987 rpcz_store.cc:276] 0807 11:10:19.739213 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:19.768138 (+ 28925us) service_pool.cc:227] Handling call
0807 11:10:19.921231 (+153093us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b5800000f98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc461a2b0 00007f3fc46057f9 00007f3fc46052a2 00007f3fbdceb684
0807 11:10:19.921464 (+ 233us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:20.538901 (+617437us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:20.915417 (+376516us) spinlock_profiling.cc:243] Waited 559 ms on lock 0x7b54006b26d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:20.915575 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1326282368}
I20260807 11:10:20.874553 28059 consensus_queue.cc:1046] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260807 11:10:20.881699 28283 raft_consensus.cc:3037] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:20.806811 28242 consensus_queue.cc:260] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.904711 28058 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:20.905462 28175 raft_consensus.cc:3032] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:20.881264 27960 consensus_queue.cc:1035] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260807 11:10:20.905015 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:20.920013 26039 raft_consensus.cc:2454] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
W20260807 11:10:20.924311 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 309487us
I20260807 11:10:20.924719 26033 raft_consensus.cc:2454] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:20.876412 28205 raft_consensus.cc:2726] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:20.898336 25806 raft_consensus.cc:2454] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
W20260807 11:10:20.926214 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 241563us
I20260807 11:10:20.910908 25981 leader_election.cc:302] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:20.927299 26936 leader_election.cc:302] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:20.927971 25872 leader_election.cc:302] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:20.908794 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:20.928514 25981 leader_election.cc:302] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:20.914055 28065 leader_election.cc:288] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:20.917291 28206 raft_consensus.cc:2726] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:20.775772 28141 consensus_queue.cc:237] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:20.945400 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:20.909444 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:20.870117 28094 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:20.920876 28294 raft_consensus.cc:3037] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:20.920819 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:20.921217 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:20.903936 25752 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:20.925171 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:20.930066 25926 raft_consensus.cc:2379] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 4: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:20.925901 28059 raft_consensus.cc:3032] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:20.931078 28298 raft_consensus.cc:2781] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:20.926462 28201 raft_consensus.cc:2726] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:20.862756 28280 raft_consensus.cc:504] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:20.945461 25917 raft_consensus.cc:1226] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:20.950971 25750 leader_election.cc:302] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:20.953471 26037 raft_consensus.cc:2932] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:20.957504 28094 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:20.958920 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:10:20.964424 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 60531.2us
I20260807 11:10:20.925397 28175 raft_consensus.cc:729] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:20.961656 27817 raft_consensus.cc:2726] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:20.936450 25981 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:20.944973 28058 raft_consensus.cc:2726] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:20.974154 25916 raft_consensus.cc:3037] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:20.992902 26975 raft_consensus.cc:1226] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:20.995309 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:20.979856 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.988585 28220 raft_consensus.cc:2781] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:20.997262 28220 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:20.997747 28220 raft_consensus.cc:3037] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:20.998247 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:20.999024 25808 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:20.980202 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.980856 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.972339 28293 raft_consensus.cc:2726] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:20.982075 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.982864 28298 raft_consensus.cc:686] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:21.006654 28298 consensus_queue.cc:237] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:20.983358 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:21.008746 25801 raft_consensus.cc:3037] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:20.965564 28059 raft_consensus.cc:729] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:20.984650 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.984685 28094 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:21.016355 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:20.986555 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.986986 28163 raft_consensus.cc:2726] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:21.019829 25806 raft_consensus.cc:1226] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:20.965806 28257 raft_consensus.cc:482] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:20.987826 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:20.989035 28137 consensus_queue.cc:1035] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:10:20.990262 28065 raft_consensus.cc:979] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815: : Instructing follower 4fd6381544574ddca559ce63b9243efc to start an election
I20260807 11:10:20.947327 28344 raft_consensus.cc:2726] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:20.974665 25752 leader_election.cc:302] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:21.014037 25800 raft_consensus.cc:1226] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:21.015475 25916 raft_consensus.cc:2454] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:10:21.015635 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:20.951305 27882 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:20.926776 28246 consensus_queue.cc:1046] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260807 11:10:21.021165 25981 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
W20260807 11:10:21.066452 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 45337.8us
I20260807 11:10:21.066507 25916 raft_consensus.cc:1226] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:21.067644 25918 raft_consensus.cc:2454] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:21.022846 25808 raft_consensus.cc:2454] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:21.024432 26987 raft_consensus.cc:1226] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3. Ops: []
W20260807 11:10:21.020447 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 58353us
W20260807 11:10:21.028884 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 103593us
I20260807 11:10:21.029655 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:21.069826 26936 leader_election.cc:302] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:21.040602 26041 raft_consensus.cc:2454] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:10:21.071754 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:21.071715 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:21.068233 25981 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:21.072533 25920 raft_consensus.cc:2379] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 2: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:21.072636 25926 raft_consensus.cc:3037] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
W20260807 11:10:21.063894 28370 log.cc:924] Time spent T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.257s user 0.002s sys 0.000s
W20260807 11:10:21.095240 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 120622us
I20260807 11:10:21.097303 25752 leader_election.cc:302] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:21.025987 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:21.028158 25798 raft_consensus.cc:2454] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:21.098300 28152 raft_consensus.cc:3032] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:21.098740 28152 raft_consensus.cc:729] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:21.062392 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:21.098886 28078 raft_consensus.cc:2726] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:21.099597 28152 consensus_queue.cc:260] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:21.095935 28344 raft_consensus.cc:2726] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:21.101152 28065 raft_consensus.cc:1067] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Signalling peer 4fd6381544574ddca559ce63b9243efc to start an election
I20260807 11:10:21.102916 25752 leader_election.cc:302] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:21.104030 25752 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:21.104744 28065 raft_consensus.cc:2726] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:21.105455 28065 raft_consensus.cc:2781] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:21.105909 28065 raft_consensus.cc:482] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:21.106376 28065 raft_consensus.cc:3037] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:21.106177 25752 leader_election.cc:302] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:21.107559 28246 raft_consensus.cc:2781] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:21.107936 28246 raft_consensus.cc:482] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:21.108530 28246 raft_consensus.cc:3037] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:21.109982 25752 leader_election.cc:302] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:21.111423 25752 leader_election.cc:302] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:21.105921 28206 raft_consensus.cc:2726] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260807 11:10:21.120460 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 51095.5us
W20260807 11:10:21.120908 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 48850.5us
I20260807 11:10:21.071720 28058 raft_consensus.cc:2726] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:21.117623 28250 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:21.117635 28152 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:21.135967 28078 raft_consensus.cc:2726] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:21.063153 28257 raft_consensus.cc:504] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:21.147403 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:21.148850 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:21.149083 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:21.094359 28072 raft_consensus.cc:2781] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:21.182590 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:21.151875 25806 raft_consensus.cc:1226] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:21.152494 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:21.155592 25872 leader_election.cc:302] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:21.186148 25872 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:20.970774 28198 consensus_queue.cc:260] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:21.170715 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:21.188993 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:21.189996 28206 consensus_queue.cc:1046] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:10:21.193581 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:21.194258 26046 raft_consensus.cc:2454] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:20.988574 28175 consensus_queue.cc:260] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:21.142771 28372 log.cc:924] Time spent T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.111s user 0.001s sys 0.000s
I20260807 11:10:21.195729 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:21.071843 28102 consensus_queue.cc:1046] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.071s
I20260807 11:10:21.178177 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:21.183982 25800 raft_consensus.cc:2379] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 2: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:21.114454 28205 raft_consensus.cc:2726] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:21.169284 28288 consensus_queue.cc:1035] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
W20260807 11:10:21.213647 28375 log.cc:924] Time spent T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
W20260807 11:10:21.213881 28377 log.cc:924] Time spent T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
W20260807 11:10:21.214877 28229 consensus_meta.cc:287] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.624s user 0.007s sys 0.004s
I20260807 11:10:21.215444 26986 raft_consensus.cc:3037] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:21.216122 28280 leader_election.cc:288] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:21.216351 28102 raft_consensus.cc:3032] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:21.216862 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:21.216928 28102 raft_consensus.cc:729] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:21.218086 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:21.218286 28102 consensus_queue.cc:260] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:21.196605 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:21.219929 28102 raft_consensus.cc:3037] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:21.109660 28059 consensus_queue.cc:260] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:21.221808 28059 raft_consensus.cc:3037] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:21.223059 26035 raft_consensus.cc:1226] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:21.224694 28257 leader_election.cc:288] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:10:21.226596 28365 log.cc:924] Time spent T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.328s user 0.000s sys 0.001s
I20260807 11:10:21.177620 25921 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f"
dest_uuid: "4fd6381544574ddca559ce63b9243efc"
from {username='slave'} at 127.0.0.1:39742
I20260807 11:10:21.229589 25921 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:10:21.230096 25921 raft_consensus.cc:3037] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:21.231205 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:21.109719 28293 consensus_queue.cc:1159] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4fd6381544574ddca559ce63b9243efc" responder_term: 3 status { last_received { term: 3 index: 6 } last_committed_idx: 5 last_received_current_leader { term: 3 index: 6 } }
I20260807 11:10:21.203857 28125 raft_consensus.cc:2781] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:21.232990 28125 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:21.233649 28125 raft_consensus.cc:3037] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:21.196161 26979 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:21.238555 28039 raft_consensus.cc:504] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:21.243142 28119 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:21.197799 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:21.198235 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:21.197479 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:10:21.247021 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 59805.5us
I20260807 11:10:21.206921 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:21.247970 26033 raft_consensus.cc:3037] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:21.207708 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:21.230530 25658 catalog_manager.cc:5495] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 1 to 2, leader changed from b22d69db45194908ad3d345eba9badd9 (127.23.120.131) to eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132). New cstate: current_term: 2 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
W20260807 11:10:21.198771 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 130613us
I20260807 11:10:21.250103 25981 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:21.254565 26986 raft_consensus.cc:1261] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 3. (term mismatch)
I20260807 11:10:21.255046 26986 pending_rounds.cc:76] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543: Aborting all ops after (but not including) 2
I20260807 11:10:21.255329 26986 pending_rounds.cc:97] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 1.3
I20260807 11:10:21.258785 25806 raft_consensus.cc:2379] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 4: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:21.261039 25872 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:21.266520 25932 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2"
dest_uuid: "4fd6381544574ddca559ce63b9243efc"
mode: GRACEFUL
new_leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
from {username='slave'} at 127.0.0.1:34992
I20260807 11:10:21.267616 25932 raft_consensus.cc:595] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Received request to transfer leadership to TS eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:21.268157 25932 raft_consensus.cc:603] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 11:10:21.265750 26042 raft_consensus.cc:2354] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 2. Current term is 3.
I20260807 11:10:21.265265 26986 raft_consensus.cc:2894] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted by new leader
I20260807 11:10:21.195885 28122 raft_consensus.cc:2781] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:21.272797 26986 raft_consensus.cc:2866] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:10:21.272480 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:21.277938 25806 raft_consensus.cc:1261] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260807 11:10:21.249769 25658 catalog_manager.cc:5495] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 1 to 2, leader changed from 606d49e053a346e486f48cb31832b815 (127.23.120.129) to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 2 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 11:10:21.215480 28229 raft_consensus.cc:2726] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:21.285849 26987 raft_consensus.cc:1226] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:21.203896 28072 raft_consensus.cc:686] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:21.287108 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:21.292114 25797 raft_consensus.cc:1226] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:21.292415 25806 raft_consensus.cc:2454] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:21.172577 28065 raft_consensus.cc:504] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:21.290444 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:21.283480 25926 raft_consensus.cc:2454] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
W20260807 11:10:21.245514 28178 consensus_queue.cc:1159] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4fd6381544574ddca559ce63b9243efc" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20260807 11:10:21.298267 25917 raft_consensus.cc:2454] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
W20260807 11:10:21.259100 27960 consensus_queue.cc:1159] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b22d69db45194908ad3d345eba9badd9" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20260807 11:10:21.348425 25801 raft_consensus.cc:2454] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:21.314334 25658 catalog_manager.cc:5495] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 1 to 2, leader changed from b22d69db45194908ad3d345eba9badd9 (127.23.120.131) to eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132). New cstate: current_term: 2 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
W20260807 11:10:21.375236 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 254158us
I20260807 11:10:21.375644 25918 raft_consensus.cc:2454] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:21.243201 28192 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:21.265982 28221 consensus_queue.cc:1046] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.061s
I20260807 11:10:21.338177 25981 leader_election.cc:302] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
W20260807 11:10:21.383965 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 82639.6us
I20260807 11:10:21.291682 27927 raft_consensus.cc:2781] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:21.279186 28175 raft_consensus.cc:3037] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:21.386457 25658 catalog_manager.cc:5495] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 1 to 2, leader changed from b22d69db45194908ad3d345eba9badd9 (127.23.120.131) to eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132). New cstate: current_term: 2 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:21.388342 25752 leader_election.cc:302] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:21.408666 25658 catalog_manager.cc:5495] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 1 to 2, leader changed from 606d49e053a346e486f48cb31832b815 (127.23.120.129) to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 2 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } }
I20260807 11:10:21.396122 26046 raft_consensus.cc:2454] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:21.397567 25786 raft_consensus.cc:1226] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:21.386274 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:21.343792 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:21.412331 25918 raft_consensus.cc:1261] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260807 11:10:21.399016 25869 leader_election.cc:302] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:21.297870 28122 raft_consensus.cc:686] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
W20260807 11:10:21.412276 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 74210.1us
I20260807 11:10:21.399179 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:21.415110 25981 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:21.416560 25981 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:21.418530 25752 leader_election.cc:302] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:21.419798 25752 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:21.420028 28198 raft_consensus.cc:3037] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:21.422394 28078 raft_consensus.cc:2726] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:21.423238 28078 raft_consensus.cc:2726] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:21.423599 28258 raft_consensus.cc:2781] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:21.424273 28078 raft_consensus.cc:2726] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:21.424844 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:21.425366 28280 raft_consensus.cc:2726] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:21.426051 28257 raft_consensus.cc:2726] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:21.426776 28201 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:21.428414 25981 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
W20260807 11:10:21.430084 28379 log.cc:924] Time spent T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.147s user 0.001s sys 0.000s
I20260807 11:10:21.429633 25658 catalog_manager.cc:5495] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 1 to 2, leader changed from b22d69db45194908ad3d345eba9badd9 (127.23.120.131) to eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132). New cstate: current_term: 2 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:21.431092 25752 leader_election.cc:302] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:21.434087 28201 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:21.461422 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 172416us
I20260807 11:10:21.438723 25658 catalog_manager.cc:5495] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 1 to 2, leader changed from 606d49e053a346e486f48cb31832b815 (127.23.120.129) to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 2 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 11:10:21.436251 25981 leader_election.cc:302] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
W20260807 11:10:21.584172 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 147995us
I20260807 11:10:21.585758 25981 leader_election.cc:302] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:21.586772 25981 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:21.436342 25872 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:21.441220 28387 raft_consensus.cc:2726] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:21.408645 27927 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:21.435062 28385 raft_consensus.cc:2726] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:21.470613 25797 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542"
dest_uuid: "606d49e053a346e486f48cb31832b815"
mode: GRACEFUL
new_leader_uuid: "4fd6381544574ddca559ce63b9243efc"
from {username='slave'} at 127.0.0.1:58114
I20260807 11:10:21.593273 25797 raft_consensus.cc:595] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Received request to transfer leadership to TS 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:21.415210 28039 leader_election.cc:288] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:21.473773 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:21.438014 28221 consensus_queue.cc:1225] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Peer 4fd6381544574ddca559ce63b9243efc log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260807 11:10:21.467821 28381 log.cc:924] Time spent T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.513s user 0.001s sys 0.002s
I20260807 11:10:21.437793 28257 raft_consensus.cc:3037] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:21.411381 28141 consensus_queue.cc:1035] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20260807 11:10:21.518013 28065 leader_election.cc:288] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:21.478045 28294 consensus_meta.cc:287] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.509s user 0.010s sys 0.003s
I20260807 11:10:21.464397 28354 consensus_queue.cc:1046] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260807 11:10:21.626168 26039 raft_consensus.cc:1261] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260807 11:10:21.495584 28389 log.cc:924] Time spent T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.061s user 0.001s sys 0.001s
I20260807 11:10:21.494871 28122 consensus_queue.cc:237] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:21.496500 28384 log.cc:924] Time spent T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.103s user 0.001s sys 0.000s
I20260807 11:10:21.493322 28072 consensus_queue.cc:237] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:21.498898 28192 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:21.503813 28201 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:21.506740 25869 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:21.476467 28205 consensus_queue.cc:1046] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260807 11:10:21.550824 25752 leader_election.cc:398] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Vote denied by peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009) with higher term. Message: Invalid argument: T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 2. Current term is 3.
I20260807 11:10:21.549898 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:21.638353 25752 leader_election.cc:401] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260807 11:10:21.519629 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:21.561069 28258 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:21.573031 28242 raft_consensus.cc:3037] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
W20260807 11:10:21.545892 28283 consensus_meta.cc:287] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.623s user 0.003s sys 0.009s
I20260807 11:10:21.398739 25931 raft_consensus.cc:1261] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260807 11:10:21.587882 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 173496us
W20260807 11:10:21.633697 28220 consensus_meta.cc:287] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.635s user 0.011s sys 0.003s
W20260807 11:10:21.653074 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 102527us
I20260807 11:10:21.654604 25752 leader_election.cc:302] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:21.596025 26033 raft_consensus.cc:2454] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:21.656363 26033 raft_consensus.cc:1226] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 1. Current term is 2. Ops: []
W20260807 11:10:21.657161 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 150555us
I20260807 11:10:21.657765 26042 raft_consensus.cc:1261] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:10:21.658061 25869 leader_election.cc:302] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:21.659504 25869 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:21.598316 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:21.529232 28110 consensus_meta.cc:287] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.605s user 0.010s sys 0.001s
I20260807 11:10:21.603564 25981 leader_election.cc:302] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
W20260807 11:10:21.604295 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 265605us
W20260807 11:10:21.663767 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 60304.5us
I20260807 11:10:21.626224 28294 raft_consensus.cc:2726] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:21.579872 28339 raft_consensus.cc:2726] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:21.591147 28055 raft_consensus.cc:2781] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:21.694060 28055 raft_consensus.cc:686] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:21.643494 28258 raft_consensus.cc:3037] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:21.622345 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:21.593737 28250 raft_consensus.cc:2781] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:21.702649 25752 leader_election.cc:302] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:21.627561 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:21.621569 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:21.630702 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260807 11:10:21.608482 25921 raft_consensus.cc:504] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:21.599121 27927 raft_consensus.cc:3037] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:21.599277 28178 raft_consensus.cc:2781] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:21.504577 28125 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:21.601562 28175 raft_consensus.cc:2781] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:21.639664 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:21.595857 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:21.531477 28246 raft_consensus.cc:504] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:21.593037 28280 consensus_queue.cc:1046] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260807 11:10:21.681981 25797 raft_consensus.cc:1261] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260807 11:10:21.682386 28395 log.cc:924] Time spent T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20260807 11:10:21.685243 25925 raft_consensus.cc:1261] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260807 11:10:21.689597 28088 raft_consensus.cc:2726] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:21.696587 25926 raft_consensus.cc:3037] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:21.699321 25796 raft_consensus.cc:2454] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:21.706290 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:10:21.705358 25917 raft_consensus.cc:1226] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:21.748276 25981 leader_election.cc:302] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:21.695200 28055 consensus_queue.cc:237] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:21.760105 25932 raft_consensus.cc:1226] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:21.761307 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:10:21.670655 28158 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:21.766379 28158 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:21.668515 28201 raft_consensus.cc:2726] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:21.754855 25920 raft_consensus.cc:1226] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:21.699965 28273 raft_consensus.cc:2726] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:21.699770 28030 raft_consensus.cc:2726] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:21.719117 28178 raft_consensus.cc:482] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:21.757287 25916 raft_consensus.cc:1261] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 11:10:21.758937 26033 raft_consensus.cc:3037] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:21.700989 28102 raft_consensus.cc:3037] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:21.742847 28244 raft_consensus.cc:2726] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:21.771894 25806 raft_consensus.cc:1226] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:21.761947 25981 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:21.751540 28250 raft_consensus.cc:686] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:21.787180 28250 consensus_queue.cc:237] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:21.771245 28058 consensus_queue.cc:1046] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20260807 11:10:21.753652 28290 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
W20260807 11:10:21.763458 28152 consensus_meta.cc:287] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.594s user 0.011s sys 0.001s
I20260807 11:10:21.796025 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:21.668805 28220 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:21.812099 26038 raft_consensus.cc:1226] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:21.696921 28192 consensus_queue.cc:1035] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.113s
I20260807 11:10:21.700136 28206 raft_consensus.cc:2726] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 11:10:21.665342 28283 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:21.818073 26039 raft_consensus.cc:1226] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:21.819273 25926 raft_consensus.cc:2454] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:21.824331 25981 leader_election.cc:302] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:21.830816 26033 raft_consensus.cc:1261] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 11:10:21.762140 28039 raft_consensus.cc:2726] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260807 11:10:21.857985 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 64434.9us
I20260807 11:10:21.778506 28178 raft_consensus.cc:3037] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:21.762545 28175 raft_consensus.cc:686] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:21.801874 28059 raft_consensus.cc:2726] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:21.828606 28058 consensus_queue.cc:1225] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer eb1fca62dcb34ed9b32188ccb19f0543 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260807 11:10:21.854434 28381 log.cc:924] Time spent T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20260807 11:10:21.797802 28246 leader_election.cc:288] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:21.860579 28192 raft_consensus.cc:2781] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:21.759498 28404 consensus_queue.cc:1035] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.153s
I20260807 11:10:21.919785 28298 consensus_queue.cc:1035] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 11:10:21.865358 28257 raft_consensus.cc:2726] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:21.920583 28141 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
W20260807 11:10:21.938822 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 261149us
I20260807 11:10:21.939929 25872 leader_election.cc:302] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:21.928577 26987 raft_consensus.cc:1261] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260807 11:10:21.832159 28221 consensus_queue.cc:1035] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.093s
I20260807 11:10:21.906850 28201 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:21.907130 28280 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:21.954610 26936 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
W20260807 11:10:21.956640 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 258681us
I20260807 11:10:21.827800 28290 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:21.961326 25869 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:21.945973 26979 consensus_meta.cc:287] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.705s user 0.012s sys 0.001s
I20260807 11:10:21.965574 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:21.964502 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:21.971625 25869 leader_election.cc:302] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:22.008417 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 44555us
W20260807 11:10:22.015450 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 60936.3us
I20260807 11:10:22.016512 26936 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:22.019528 26979 raft_consensus.cc:1261] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 11:10:21.950309 28106 raft_consensus.cc:2781] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election won for term 2
W20260807 11:10:22.024818 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 51548.4us
W20260807 11:10:22.027648 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 56114.8us
I20260807 11:10:22.030534 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:22.033209 28244 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:22.036818 25799 raft_consensus.cc:1226] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260807 11:10:22.037385 28354 raft_consensus.cc:2726] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 11:10:22.037077 28244 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:22.039047 28414 log.cc:924] Time spent T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20260807 11:10:22.039675 28354 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:22.041508 28421 consensus_queue.cc:1035] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 11:10:22.044705 26034 raft_consensus.cc:1261] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:10:21.946179 28141 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:22.006611 28215 raft_consensus.cc:2781] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:22.033419 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:22.051225 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:21.950865 28201 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:22.051465 26042 raft_consensus.cc:3037] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:21.952126 28280 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:22.051733 25803 raft_consensus.cc:1261] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:10:21.959360 28116 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:22.018740 28125 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:22.086620 27960 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:21.962515 28252 consensus_queue.cc:1035] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.089s
I20260807 11:10:22.087236 27960 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:22.087919 28392 log.cc:924] Time spent T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.163s user 0.000s sys 0.000s
I20260807 11:10:22.081867 28289 consensus_queue.cc:1046] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260807 11:10:22.009219 28425 raft_consensus.cc:2726] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:22.055586 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:22.059557 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:22.053045 25981 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:22.029528 25921 leader_election.cc:288] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:21.962055 27927 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:22.023321 28298 raft_consensus.cc:2726] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:22.085387 28116 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:22.040279 28354 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:22.113524 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:10:22.093237 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 45811.1us
I20260807 11:10:22.039669 28244 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:22.052588 28215 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:22.130119 28215 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:21.966207 28229 consensus_queue.cc:1046] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.097s
I20260807 11:10:22.063902 28257 consensus_queue.cc:1046] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20260807 11:10:22.065896 28283 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:22.053056 28220 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:22.090930 28059 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:22.108676 28416 log.cc:924] Time spent T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.136s user 0.000s sys 0.001s
I20260807 11:10:21.967696 28175 consensus_queue.cc:237] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:22.131735 25919 raft_consensus.cc:1226] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:22.092959 28106 raft_consensus.cc:686] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:22.103534 28055 raft_consensus.cc:2726] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 11:10:22.122725 25981 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
W20260807 11:10:22.148790 28406 log.cc:924] Time spent T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.196s user 0.001s sys 0.001s
I20260807 11:10:22.105615 27817 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
W20260807 11:10:22.167639 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 45011.3us
I20260807 11:10:22.185763 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:22.186477 25797 raft_consensus.cc:2454] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:22.188123 25981 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:22.051404 28426 consensus_queue.cc:1046] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.095s
I20260807 11:10:22.110859 28030 consensus_queue.cc:1035] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
W20260807 11:10:22.210916 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 182476us
I20260807 11:10:22.094627 28192 raft_consensus.cc:686] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:22.150192 28246 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:22.251514 25800 raft_consensus.cc:1261] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 11:10:22.256925 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:22.190153 28257 raft_consensus.cc:2726] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:22.230903 28055 raft_consensus.cc:2781] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:22.210455 27960 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:22.262570 25803 raft_consensus.cc:1261] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 11:10:22.166370 28059 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:22.312776 25926 raft_consensus.cc:1261] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260807 11:10:22.316991 26046 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.038s sys 0.067s
I20260807 11:10:22.323642 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:22.264703 28055 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 11:10:22.321539 26046 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.575s user 0.038s sys 0.067s
I20260807 11:10:22.329797 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:22.256704 28201 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:22.212752 27817 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:22.342828 25797 raft_consensus.cc:1261] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:10:22.257798 28229 raft_consensus.cc:482] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:22.261621 28289 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:22.347824 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1372) took 1074 ms. Trace:
I20260807 11:10:22.348357 26041 rpcz_store.cc:276] 0807 11:10:21.273802 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:21.273937 (+ 135us) service_pool.cc:227] Handling call
0807 11:10:21.274426 (+ 489us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:21.274790 (+ 364us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:21.274814 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:21.377453 (+102639us) log.cc:842] Serialized 572 byte log entry
0807 11:10:21.625887 (+248434us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:21.625921 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:21.625982 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:22.347620 (+721638us) raft_consensus.cc:1671] finished
0807 11:10:22.347649 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:22.347770 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":217656,"threads_started":1,"wal-append.queue_time_us":525814}
I20260807 11:10:22.252143 28094 consensus_queue.cc:1046] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.083s
I20260807 11:10:22.354657 26042 raft_consensus.cc:2454] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:22.308944 28290 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:22.257496 28246 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:22.337653 28055 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
W20260807 11:10:22.371695 28434 log.cc:924] Time spent T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.124s user 0.001s sys 0.000s
I20260807 11:10:22.371805 28059 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:22.373558 28116 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:22.351154 28229 raft_consensus.cc:504] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:22.361480 25798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1437) took 1180 ms. Trace:
I20260807 11:10:22.380085 25798 rpcz_store.cc:276] 0807 11:10:21.171011 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:21.218977 (+ 47966us) service_pool.cc:227] Handling call
0807 11:10:21.221741 (+ 2764us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:21.222426 (+ 685us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:21.222465 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:21.470862 (+248397us) log.cc:842] Serialized 572 byte log entry
0807 11:10:21.764360 (+293498us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:21.764408 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:21.764489 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:22.335514 (+571025us) raft_consensus.cc:1671] finished
0807 11:10:22.341697 (+ 6183us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:22.351177 (+ 9480us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":236943,"threads_started":1,"wal-append.queue_time_us":446666}
I20260807 11:10:22.357695 28435 consensus_queue.cc:1035] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.031s
I20260807 11:10:22.358613 28220 consensus_queue.cc:1035] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20260807 11:10:22.354126 28289 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:22.387076 28065 raft_consensus.cc:3032] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:22.387549 28065 raft_consensus.cc:729] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:22.387706 28289 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:22.388572 25921 raft_consensus.cc:1261] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 11:10:22.390170 28246 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:22.361789 28412 log.cc:924] Time spent T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.351s user 0.000s sys 0.002s
I20260807 11:10:22.395758 28229 leader_election.cc:288] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:22.398947 28215 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:22.405076 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:22.405768 26039 raft_consensus.cc:2454] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:22.407142 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:22.129601 27882 raft_consensus.cc:2932] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:22.407104 25752 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:22.363411 28141 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:22.410550 28436 log.cc:924] Time spent T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
W20260807 11:10:22.411021 28424 log.cc:924] Time spent T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.361s user 0.001s sys 0.000s
W20260807 11:10:22.430676 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 189380us
I20260807 11:10:22.320960 28192 consensus_queue.cc:237] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:22.368320 28280 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:22.116503 28252 raft_consensus.cc:2932] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:22.411198 28059 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:22.422667 28311 consensus_queue.cc:1035] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 11:10:22.465891 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:22.411441 27927 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:22.467124 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:22.433538 28152 raft_consensus.cc:2781] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:22.472786 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:22.475490 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:22.476181 26986 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:22.358027 28106 consensus_queue.cc:237] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:22.477512 25981 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:22.484788 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:22.468816 28102 consensus_meta.cc:287] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.675s user 0.011s sys 0.007s
I20260807 11:10:22.490691 26936 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:22.464365 28432 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:22.493543 26046 raft_consensus.cc:3037] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
W20260807 11:10:22.477849 28427 log.cc:924] Time spent T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.333s user 0.000s sys 0.001s
I20260807 11:10:22.474843 28152 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:22.472923 28354 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:22.388702 28065 consensus_queue.cc:260] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:22.467355 28141 consensus_queue.cc:1035] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260807 11:10:22.443666 28423 consensus_queue.cc:1035] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260807 11:10:22.509661 26039 raft_consensus.cc:1226] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:22.469710 28059 raft_consensus.cc:504] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:22.511142 25916 raft_consensus.cc:3032] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:22.526400 25916 raft_consensus.cc:729] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:22.512455 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:22.527959 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:22.512787 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:22.529227 25926 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
W20260807 11:10:22.518052 28428 log.cc:924] Time spent T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.343s user 0.000s sys 0.001s
I20260807 11:10:22.500829 28280 raft_consensus.cc:2781] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:22.524441 26046 raft_consensus.cc:1261] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 11:10:22.484015 28215 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:22.540081 27960 consensus_queue.cc:1046] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260807 11:10:22.547255 28158 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:22.548656 27817 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:22.525197 28198 consensus_meta.cc:287] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 1.104s user 0.008s sys 0.008s
I20260807 11:10:22.549862 28078 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:22.550549 26037 raft_consensus.cc:3032] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:22.551126 26037 raft_consensus.cc:729] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:22.507457 28152 raft_consensus.cc:3037] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:22.571098 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1440) took 1078 ms. Trace:
I20260807 11:10:22.571571 25917 rpcz_store.cc:276] 0807 11:10:21.493198 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:21.493742 (+ 544us) service_pool.cc:227] Handling call
0807 11:10:21.494033 (+ 291us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:21.760897 (+266864us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:22.570823 (+809926us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b5400185e54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:22.571029 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":288269184}
I20260807 11:10:22.505378 28432 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:22.577416 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1437) took 1172 ms. Trace:
I20260807 11:10:22.577909 25932 rpcz_store.cc:276] 0807 11:10:21.405333 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:21.405487 (+ 154us) service_pool.cc:227] Handling call
0807 11:10:21.410389 (+ 4902us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:21.764731 (+354342us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:22.577155 (+812424us) spinlock_profiling.cc:243] Waited 349 ms on lock 0x7b5400270954. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:22.577359 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":803688704}
I20260807 11:10:22.539799 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:22.581722 25801 raft_consensus.cc:3032] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:22.582129 25801 raft_consensus.cc:729] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:22.582962 25801 consensus_queue.cc:260] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:22.538384 26938 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:22.584582 25806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1445) took 1291 ms. Trace:
W20260807 11:10:22.584738 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 46492.8us
I20260807 11:10:22.584998 25806 rpcz_store.cc:276] 0807 11:10:21.292899 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:21.293216 (+ 317us) service_pool.cc:227] Handling call
0807 11:10:21.299048 (+ 5832us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:21.777850 (+478802us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:22.584331 (+806481us) spinlock_profiling.cc:243] Waited 473 ms on lock 0x7b5400137254. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:22.584523 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1086803200}
I20260807 11:10:22.586373 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:22.496352 28102 raft_consensus.cc:2726] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009) with higher term. Message: Invalid argument: T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 2. Current term is 3.
I20260807 11:10:22.542889 25916 consensus_queue.cc:260] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:22.555097 28447 log.cc:924] Time spent T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.088s user 0.001s sys 0.000s
I20260807 11:10:22.588089 25806 raft_consensus.cc:1226] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
W20260807 11:10:22.556171 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 54944.4us
I20260807 11:10:22.588807 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:22.591255 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:22.556533 28059 leader_election.cc:288] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:22.509465 28455 raft_consensus.cc:2726] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:22.596952 26046 raft_consensus.cc:1261] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 11:10:22.557049 28061 consensus_queue.cc:1035] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 11:10:22.580754 25921 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:22.558743 25920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1436) took 1310 ms. Trace:
I20260807 11:10:22.512151 28065 raft_consensus.cc:3037] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:22.542740 25981 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
W20260807 11:10:22.553879 28178 consensus_meta.cc:287] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.639s user 0.005s sys 0.011s
I20260807 11:10:22.550500 28078 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:22.562248 25919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1439) took 1086 ms. Trace:
I20260807 11:10:22.611192 25919 rpcz_store.cc:276] 0807 11:10:21.475625 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:21.477808 (+ 2183us) service_pool.cc:227] Handling call
0807 11:10:21.478110 (+ 302us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:22.163954 (+685844us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:22.561978 (+398024us) spinlock_profiling.cc:243] Waited 653 ms on lock 0x7b54003ff7d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:22.562187 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1502740736}
I20260807 11:10:22.563547 25926 raft_consensus.cc:2454] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:10:22.612344 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:22.612986 25932 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:22.552040 26037 consensus_queue.cc:260] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:22.567901 28425 raft_consensus.cc:3032] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:22.551576 28280 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:22.576352 28432 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:22.600595 25920 rpcz_store.cc:276] 0807 11:10:21.248639 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:21.248917 (+ 278us) service_pool.cc:227] Handling call
0807 11:10:21.281551 (+ 32634us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:21.768028 (+486477us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:22.558438 (+790410us) spinlock_profiling.cc:243] Waited 473 ms on lock 0x7b54003b29d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:22.558659 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1087917440}
I20260807 11:10:22.612690 26041 raft_consensus.cc:2932] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:22.634531 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:22.635891 26038 raft_consensus.cc:1261] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 11:10:22.636989 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:22.603006 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:22.639408 25803 raft_consensus.cc:2396] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260807 11:10:22.603452 25801 raft_consensus.cc:3037] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:22.607668 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:22.609818 26046 raft_consensus.cc:1261] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:10:22.645468 25917 raft_consensus.cc:2396] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260807 11:10:22.645865 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:22.629447 26037 raft_consensus.cc:3037] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
W20260807 11:10:22.630594 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 88035.9us
I20260807 11:10:22.659358 26034 raft_consensus.cc:1226] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:22.602303 28198 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:22.670821 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:22.634785 28280 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:22.632786 28293 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:22.650543 28258 consensus_meta.cc:287] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.956s user 0.013s sys 0.004s
I20260807 11:10:22.664880 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:22.689607 25917 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:22.677466 25872 leader_election.cc:302] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:22.678617 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:22.679337 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:22.636160 28058 raft_consensus.cc:2726] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:22.633471 28425 raft_consensus.cc:729] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:22.685379 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:22.681411 26037 raft_consensus.cc:1261] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260807 11:10:22.696013 25801 raft_consensus.cc:2454] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:22.692731 26033 raft_consensus.cc:2454] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:22.712002 26037 pending_rounds.cc:76] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9: Aborting all ops after (but not including) 4
I20260807 11:10:22.712371 26037 pending_rounds.cc:97] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260807 11:10:22.712664 26037 raft_consensus.cc:2866] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:10:22.659349 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:22.673956 28056 raft_consensus.cc:3037] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:22.637612 28178 raft_consensus.cc:504] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:22.610455 28158 consensus_queue.cc:1035] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
I20260807 11:10:22.721963 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:22.735448 25926 raft_consensus.cc:2454] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
W20260807 11:10:22.727210 27960 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
W20260807 11:10:22.728022 28385 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260807 11:10:22.673741 28432 consensus_queue.cc:1035] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20260807 11:10:22.753789 28055 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:22.725001 28457 raft_consensus.cc:2781] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:22.755084 25872 leader_election.cc:302] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:22.683362 28198 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:22.706755 28293 raft_consensus.cc:504] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:22.707420 28258 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:22.723527 27927 consensus_queue.cc:1046] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260807 11:10:22.761636 28398 raft_consensus.cc:2726] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:22.768044 28143 raft_consensus.cc:2781] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:22.779640 25803 raft_consensus.cc:2454] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:22.782795 28198 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:22.789166 28198 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:22.789892 28198 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:22.778699 28058 raft_consensus.cc:482] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:22.792505 28058 raft_consensus.cc:504] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:22.767622 28457 raft_consensus.cc:686] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:22.794034 28178 leader_election.cc:288] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:22.795800 28058 leader_election.cc:288] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:22.795145 28457 consensus_queue.cc:237] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:22.738619 28354 raft_consensus.cc:482] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:22.797417 28354 raft_consensus.cc:504] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:22.756903 28426 consensus_queue.cc:1035] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
W20260807 11:10:22.774296 28242 consensus_meta.cc:287] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 1.111s user 0.018s sys 0.000s
I20260807 11:10:22.799026 28143 raft_consensus.cc:482] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:22.799154 26038 raft_consensus.cc:2932] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:22.808593 25786 raft_consensus.cc:2454] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:22.820173 28143 raft_consensus.cc:3037] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:22.822893 28055 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:22.846352 28293 leader_election.cc:288] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:22.772598 28425 consensus_queue.cc:260] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:22.853495 28229 consensus_queue.cc:1046] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:22.775669 28456 consensus_queue.cc:1035] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.174s
I20260807 11:10:22.846967 28152 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:22.873092 28258 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:22.851468 28280 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:22.884418 25808 raft_consensus.cc:2454] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:22.906433 28339 raft_consensus.cc:3032] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Stepping down as leader of term 1
W20260807 11:10:22.906754 28460 log.cc:924] Time spent T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.316s user 0.000s sys 0.005s
I20260807 11:10:22.906916 28339 raft_consensus.cc:729] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:22.908161 28339 consensus_queue.cc:260] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:22.903952 28079 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:22.935753 28449 raft_consensus.cc:3032] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:22.936566 28449 raft_consensus.cc:729] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:22.937366 28273 raft_consensus.cc:3032] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:22.937829 28273 raft_consensus.cc:729] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:22.939978 25801 raft_consensus.cc:1226] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:22.910928 28079 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:22.940928 28273 consensus_queue.cc:260] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:22.942123 25796 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1351) took 1962 ms (client timeout 1639 ms). Trace:
W20260807 11:10:22.942529 25796 rpcz_store.cc:269] 0807 11:10:20.979804 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:20.980606 (+ 802us) service_pool.cc:227] Handling call
0807 11:10:22.941926 (+1961320us) spinlock_profiling.cc:243] Waited 695 ms on lock 0x7b54000e4cd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:22.942047 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1598377728}
I20260807 11:10:22.935966 28339 raft_consensus.cc:3037] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:22.935788 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:22.948978 25799 raft_consensus.cc:1261] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:10:22.971850 26039 raft_consensus.cc:2379] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 4: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
W20260807 11:10:22.971953 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 74860.1us
W20260807 11:10:22.975549 28467 log.cc:924] Time spent T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.110s user 0.000s sys 0.000s
I20260807 11:10:22.942539 28273 raft_consensus.cc:3037] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:23.021848 28152 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:23.032670 26036 raft_consensus.cc:1261] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 11:10:23.033812 28137 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:22.938277 28449 consensus_queue.cc:260] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:23.034428 28137 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:23.036100 28449 raft_consensus.cc:3037] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:23.039377 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:23.057653 25869 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.794s user 0.017s sys 0.077s
W20260807 11:10:23.058106 25869 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.794s user 0.017s sys 0.077s
I20260807 11:10:23.064911 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:23.067502 26041 raft_consensus.cc:2379] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 3: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:23.075937 28280 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:23.076833 28152 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:23.077523 28152 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:23.069396 28065 raft_consensus.cc:482] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:23.081794 28354 leader_election.cc:288] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:23.081421 28065 raft_consensus.cc:504] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:23.084949 28065 leader_election.cc:288] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:23.080744 28152 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:23.093773 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:23.094568 26041 raft_consensus.cc:2454] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:23.098790 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:23.126996 26034 raft_consensus.cc:2454] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
W20260807 11:10:23.061605 28175 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.009s sys 0.020s
I20260807 11:10:23.130270 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:23.134102 26041 raft_consensus.cc:3037] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:23.129151 27960 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:23.130812 28292 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:23.131533 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:23.158074 26033 raft_consensus.cc:2454] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:23.165158 26041 raft_consensus.cc:2454] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:23.042454 28143 raft_consensus.cc:504] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:23.092551 28065 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:23.177002 25801 raft_consensus.cc:2932] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:23.185402 28065 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:23.187299 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:23.187988 28065 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:23.188870 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:23.189594 26987 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:23.188894 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:10:23.156318 25808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1429) took 2009 ms (client timeout 2207 ms). Trace:
I20260807 11:10:23.157035 28292 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:23.171265 28425 raft_consensus.cc:3037] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:23.172339 28462 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:23.177420 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:23.203533 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:23.204407 25796 raft_consensus.cc:2379] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 2: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
W20260807 11:10:23.141054 28175 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.597s user 0.010s sys 0.021s
I20260807 11:10:23.178000 25803 raft_consensus.cc:1226] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:23.184464 28385 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:23.184682 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:23.188148 26986 raft_consensus.cc:2379] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 4: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:23.170399 27960 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:23.193148 26034 raft_consensus.cc:1226] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:23.230417 25799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1446) took 1190 ms. Trace:
I20260807 11:10:23.206069 26041 raft_consensus.cc:1226] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 2. Current term is 3. Ops: []
W20260807 11:10:23.192165 25808 rpcz_store.cc:269] 0807 11:10:21.147278 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:21.148611 (+ 1333us) service_pool.cc:227] Handling call
0807 11:10:23.156126 (+2007515us) spinlock_profiling.cc:243] Waited 1.64 s on lock 0x7b5400265554. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:23.156249 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3775930112}
I20260807 11:10:23.205883 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:23.233086 25916 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:23.217819 25786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1362) took 1624 ms. Trace:
W20260807 11:10:23.218364 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 56822.2us
I20260807 11:10:23.235181 25752 leader_election.cc:302] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:23.237103 25752 leader_election.cc:302] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:23.219142 25869 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:23.220923 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:23.227941 26036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1345) took 2939 ms. Trace:
I20260807 11:10:23.238296 26036 rpcz_store.cc:276] 0807 11:10:20.288833 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:20.299671 (+ 10838us) service_pool.cc:227] Handling call
0807 11:10:20.299951 (+ 280us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:23.033577 (+2733626us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:23.227653 (+194076us) spinlock_profiling.cc:243] Waited 2.73 s on lock 0x7b54000bb154. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:23.227870 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6283825536}
I20260807 11:10:23.230917 25799 rpcz_store.cc:276] 0807 11:10:22.039590 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:22.039744 (+ 154us) service_pool.cc:227] Handling call
0807 11:10:22.948557 (+908813us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:22.949397 (+ 840us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:23.230185 (+280788us) spinlock_profiling.cc:243] Waited 908 ms on lock 0x7b54000e4cd0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:23.230348 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2089416576}
I20260807 11:10:23.233966 25786 rpcz_store.cc:276] 0807 11:10:21.593776 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:21.593927 (+ 151us) service_pool.cc:227] Handling call
0807 11:10:23.217647 (+1623720us) spinlock_profiling.cc:243] Waited 884 ms on lock 0x7b54003ac5d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:23.217760 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2032998656}
I20260807 11:10:23.259014 25752 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:23.260288 28293 raft_consensus.cc:2781] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:23.260926 28293 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:23.261387 28293 raft_consensus.cc:3037] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:23.262918 25808 raft_consensus.cc:1226] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:23.266669 25752 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:23.269263 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:23.271996 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:23.279222 25803 raft_consensus.cc:1261] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:10:23.281041 25799 raft_consensus.cc:1261] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:10:23.189955 26976 raft_consensus.cc:2932] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:23.265147 26979 raft_consensus.cc:2454] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:23.256145 28152 raft_consensus.cc:2781] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:23.300638 25796 raft_consensus.cc:2454] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:23.277930 28242 raft_consensus.cc:2781] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:23.212711 28280 raft_consensus.cc:482] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:23.312994 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 93962.1us
I20260807 11:10:23.295893 28244 raft_consensus.cc:2781] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:23.317198 28354 raft_consensus.cc:2781] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:23.317512 28244 raft_consensus.cc:686] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:23.317756 28354 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:23.318493 28354 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:23.317800 27817 raft_consensus.cc:482] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:23.323567 27817 raft_consensus.cc:504] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:23.266418 28198 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:23.339881 28438 consensus_queue.cc:1035] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260807 11:10:23.317405 28122 raft_consensus.cc:3032] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:23.301587 28242 raft_consensus.cc:482] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:23.301270 28152 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:23.354007 27817 leader_election.cc:288] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:23.362224 28438 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:23.362907 28078 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:23.367813 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:23.368727 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:23.325704 28137 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:23.377213 26041 raft_consensus.cc:1226] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:23.319065 28244 consensus_queue.cc:237] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:23.307411 28175 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:23.384133 26034 raft_consensus.cc:2379] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 3: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:23.256522 28385 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:23.308866 27927 consensus_queue.cc:1046] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20260807 11:10:23.396147 28229 raft_consensus.cc:482] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:23.388262 27817 consensus_queue.cc:1046] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:23.225185 26980 raft_consensus.cc:2932] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:23.365742 28122 raft_consensus.cc:729] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:23.513583 28122 consensus_queue.cc:260] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:23.515466 28122 raft_consensus.cc:3037] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:23.372287 28242 raft_consensus.cc:3037] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:23.375914 28152 raft_consensus.cc:3037] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:23.374609 28205 raft_consensus.cc:482] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:23.527480 28205 raft_consensus.cc:504] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:23.327360 28280 raft_consensus.cc:504] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:23.537890 28280 leader_election.cc:288] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:23.424196 28453 log.cc:924] Time spent T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.804s user 0.006s sys 0.001s
I20260807 11:10:23.538153 28137 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:23.539124 28137 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:23.539736 28205 leader_election.cc:288] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:23.433979 28292 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:23.540045 25800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 145) took 1289 ms. Trace:
I20260807 11:10:23.541671 28137 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:23.540640 25800 rpcz_store.cc:276] 0807 11:10:22.250344 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:22.260210 (+ 9866us) service_pool.cc:227] Handling call
0807 11:10:22.260546 (+ 336us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:22.266076 (+ 5530us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:22.266150 (+ 74us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:22.266812 (+ 662us) log.cc:842] Serialized 26 byte log entry
0807 11:10:22.326898 (+ 60086us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:22.327036 (+ 138us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:22.327304 (+ 268us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:23.225295 (+897991us) spinlock_profiling.cc:243] Waited 27.1 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:10:23.539576 (+314281us) raft_consensus.cc:1671] finished
0807 11:10:23.539620 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:23.539945 (+ 325us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5006,"raft.queue_time_us":44506,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":15960,"spinlock_wait_cycles":62432640,"thread_start_us":59392,"threads_started":1,"wal-append.queue_time_us":214388}
I20260807 11:10:23.435235 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:23.544174 26036 raft_consensus.cc:2379] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 2: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:23.435317 28079 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:23.216101 28462 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:23.443607 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:23.420356 27960 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:23.457273 28339 consensus_meta.cc:287] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.513s user 0.003s sys 0.010s
I20260807 11:10:23.501281 28229 raft_consensus.cc:504] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:23.510494 27817 raft_consensus.cc:3032] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:23.553196 28137 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:23.566982 28137 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:23.570163 28137 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:23.571442 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:23.573581 27817 raft_consensus.cc:729] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:23.574918 27817 consensus_queue.cc:260] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:23.576750 27817 raft_consensus.cc:3037] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:23.554069 28462 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:23.555204 28079 raft_consensus.cc:482] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:23.581545 28079 raft_consensus.cc:504] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:23.551304 28292 raft_consensus.cc:482] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:23.563102 27960 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:23.585678 28292 raft_consensus.cc:504] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:23.585842 27960 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:23.589773 27960 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:23.565069 28229 leader_election.cc:288] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:23.590821 28292 leader_election.cc:288] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:23.592684 28385 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:23.394788 28438 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:23.612010 25916 raft_consensus.cc:1261] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:10:23.624812 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:23.625753 26034 raft_consensus.cc:2454] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:23.626854 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:23.552294 28143 leader_election.cc:288] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:23.612882 25916 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1410) took 1107 ms. Trace:
W20260807 11:10:23.643293 28286 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260807 11:10:23.498674 28175 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:23.602946 28079 leader_election.cc:288] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:23.652146 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:23.655079 26037 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:23.655697 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:23.672176 28229 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:23.673153 28229 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:23.677388 28286 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:23.678669 28286 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:23.639871 28385 raft_consensus.cc:482] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:23.646160 25916 rpcz_store.cc:276] 0807 11:10:22.505674 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:22.505825 (+ 151us) service_pool.cc:227] Handling call
0807 11:10:22.510992 (+ 5167us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:23.232799 (+721807us) spinlock_profiling.cc:243] Waited 581 ms on lock 0x7b58003caab0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf63ff63 00007f3fbf6a3aba 00007f3fbf6bb3af 00007f3fbf6bcb4e 00007f3fbf6c6c89 00007f3fbf6c7556 00007f3fbf6c3edf 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684
0807 11:10:23.612506 (+379707us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:23.612817 (+ 311us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1335380352}
I20260807 11:10:23.700388 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:23.712714 25872 leader_election.cc:302] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:23.728904 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:23.730113 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:23.746644 26034 raft_consensus.cc:2454] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:23.747802 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:23.731808 28438 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:23.750519 26037 raft_consensus.cc:2379] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 2: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:23.764962 28205 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:23.766958 28079 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:23.769204 27960 raft_consensus.cc:482] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:23.769836 28229 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:23.781587 28079 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:23.795650 28079 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:23.785239 28205 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:23.682343 28286 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:23.791138 27960 raft_consensus.cc:504] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:23.822914 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 110331us
W20260807 11:10:23.837648 28449 consensus_meta.cc:287] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.801s user 0.003s sys 0.012s
I20260807 11:10:23.832909 28438 raft_consensus.cc:2781] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260807 11:10:23.870076 28386 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.159s user 0.040s sys 0.074s
W20260807 11:10:23.870645 28386 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.165s user 0.040s sys 0.074s
W20260807 11:10:23.880216 25920 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.147s user 0.012s sys 0.060s
W20260807 11:10:23.880659 25920 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.147s user 0.012s sys 0.060s
I20260807 11:10:23.783406 28385 raft_consensus.cc:504] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:23.882038 25806 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.267s user 0.023s sys 0.058s
W20260807 11:10:23.882558 25806 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.268s user 0.023s sys 0.058s
W20260807 11:10:23.885035 25755 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.199s user 0.045s sys 0.108s
W20260807 11:10:23.885388 25755 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.200s user 0.045s sys 0.108s
W20260807 11:10:23.889346 26938 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.225s user 0.005s sys 0.014s
W20260807 11:10:23.889776 26938 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.226s user 0.005s sys 0.014s
W20260807 11:10:23.890108 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 1.22605e+06us
W20260807 11:10:23.827282 28192 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.279s user 0.022s sys 0.061s
W20260807 11:10:23.931950 28192 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.384s user 0.022s sys 0.061s
I20260807 11:10:23.962524 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:23.965438 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:23.963531 26036 raft_consensus.cc:2454] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:23.997628 26041 raft_consensus.cc:2454] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
W20260807 11:10:23.998657 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 107774us
I20260807 11:10:23.865929 28438 raft_consensus.cc:482] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 11:10:23.936363 28106 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.414s user 0.035s sys 0.097s
W20260807 11:10:23.884881 28423 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.159s user 0.032s sys 0.059s
I20260807 11:10:23.866360 28205 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:23.905757 28079 raft_consensus.cc:482] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:23.957291 28114 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.195s user 0.007s sys 0.014s
I20260807 11:10:23.934247 28078 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
W20260807 11:10:24.002761 28420 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.261s user 0.006s sys 0.014s
W20260807 11:10:24.042541 28152 consensus_meta.cc:287] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.518s user 0.004s sys 0.010s
W20260807 11:10:24.042541 28293 consensus_meta.cc:287] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.781s user 0.015s sys 0.000s
W20260807 11:10:24.057504 28425 consensus_meta.cc:287] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.840s user 0.012s sys 0.003s
I20260807 11:10:24.057881 28498 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:24.112334 26939 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.425s user 0.009s sys 0.025s
I20260807 11:10:24.076967 28292 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:24.080279 28483 raft_consensus.cc:482] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:24.094601 28143 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:24.085453 28175 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:24.108927 28286 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:24.123117 27960 leader_election.cc:288] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:24.213091 28242 consensus_meta.cc:287] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.696s user 0.014s sys 0.000s
I20260807 11:10:24.108462 28192 raft_consensus.cc:482] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:24.227466 28192 raft_consensus.cc:504] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:24.229591 28500 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:24.230126 28192 leader_election.cc:288] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:24.062723 28386 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:24.230321 28500 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:24.231328 28386 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:24.171918 28106 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.649s user 0.036s sys 0.103s
W20260807 11:10:24.218542 28354 consensus_meta.cc:287] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.899s user 0.013s sys 0.008s
I20260807 11:10:24.233265 28354 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:24.234071 28500 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195)
W20260807 11:10:24.210834 26939 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.524s user 0.010s sys 0.025s
W20260807 11:10:24.175565 28088 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.028s user 0.024s sys 0.022s
W20260807 11:10:24.235018 28088 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.090s user 0.025s sys 0.023s
W20260807 11:10:24.173843 28114 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.414s user 0.007s sys 0.015s
I20260807 11:10:24.235555 28354 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:24.220279 28143 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:24.171355 28423 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.461s user 0.032s sys 0.059s
I20260807 11:10:24.177764 28385 leader_election.cc:288] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:24.223523 28175 raft_consensus.cc:482] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:24.247367 28175 raft_consensus.cc:504] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:24.177290 28078 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:24.254583 28078 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:24.216084 28292 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:24.257359 28292 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:24.178592 28079 raft_consensus.cc:504] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:24.173868 28456 consensus_queue.cc:1035] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20260807 11:10:24.213512 28293 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:24.221485 28252 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.221s user 0.009s sys 0.009s
I20260807 11:10:24.242658 28242 raft_consensus.cc:504] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:24.214399 28498 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:24.213683 28152 raft_consensus.cc:504] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:24.217962 28483 raft_consensus.cc:504] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:24.262342 28252 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.276s user 0.009s sys 0.009s
W20260807 11:10:24.215097 28122 consensus_meta.cc:287] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.699s user 0.009s sys 0.007s
I20260807 11:10:24.171068 28438 raft_consensus.cc:3037] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
W20260807 11:10:24.175398 28457 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.299s user 0.000s sys 0.020s
I20260807 11:10:24.238088 28286 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:24.176019 28420 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.434s user 0.006s sys 0.015s
I20260807 11:10:24.223877 28449 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
W20260807 11:10:24.312496 28457 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.436s user 0.000s sys 0.020s
I20260807 11:10:24.322546 28498 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:24.323784 28483 leader_election.cc:288] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:24.357451 25797 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.619s user 0.036s sys 0.089s
W20260807 11:10:24.358428 25797 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.625s user 0.036s sys 0.089s
W20260807 11:10:24.356518 25798 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.571s user 0.018s sys 0.046s
W20260807 11:10:24.361876 25798 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.577s user 0.018s sys 0.046s
W20260807 11:10:24.356499 25750 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.717s user 0.020s sys 0.027s
W20260807 11:10:24.363795 25750 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.724s user 0.021s sys 0.027s
I20260807 11:10:24.049207 25872 leader_election.cc:302] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:10:24.366062 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 317303us
W20260807 11:10:24.366288 25801 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.149s user 0.007s sys 0.009s
W20260807 11:10:24.366930 25801 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.150s user 0.007s sys 0.009s
W20260807 11:10:24.088856 26038 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.054s user 0.006s sys 0.047s
W20260807 11:10:24.368432 26039 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.143s user 0.022s sys 0.025s
W20260807 11:10:24.368606 26038 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.334s user 0.006s sys 0.047s
W20260807 11:10:24.369057 26039 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.144s user 0.023s sys 0.025s
W20260807 11:10:24.369433 25869 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.036s user 0.003s sys 0.008s
W20260807 11:10:24.369948 25869 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.037s user 0.003s sys 0.008s
W20260807 11:10:24.213469 26042 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.503s user 0.023s sys 0.036s
I20260807 11:10:24.370317 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1403) took 1337 ms. Trace:
W20260807 11:10:24.370501 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 1.03714e+06us
W20260807 11:10:24.371003 26042 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.661s user 0.023s sys 0.037s
W20260807 11:10:24.371647 26035 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.394s user 0.013s sys 0.013s
W20260807 11:10:24.372303 26035 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.395s user 0.013s sys 0.013s
W20260807 11:10:24.372538 26046 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.879s user 0.009s sys 0.008s
I20260807 11:10:24.370859 26038 rpcz_store.cc:276] 0807 11:10:23.032698 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:23.032893 (+ 195us) service_pool.cc:227] Handling call
0807 11:10:23.033358 (+ 465us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:23.033532 (+ 174us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:23.033554 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:23.034037 (+ 483us) log.cc:842] Serialized 26 byte log entry
0807 11:10:24.369546 (+1335509us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:10:24.369653 (+ 107us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:24.369812 (+ 159us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:24.369927 (+ 115us) raft_consensus.cc:1671] finished
0807 11:10:24.369944 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:24.370184 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1334736,"threads_started":1,"wal-append.queue_time_us":1258774}
W20260807 11:10:24.372962 26936 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.686s user 0.024s sys 0.087s
W20260807 11:10:24.373169 26046 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.880s user 0.009s sys 0.009s
W20260807 11:10:24.373507 26936 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.687s user 0.025s sys 0.087s
W20260807 11:10:24.374156 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 1.69846e+06us
I20260807 11:10:24.144786 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:24.059300 25981 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.409s user 0.028s sys 0.086s
W20260807 11:10:24.445295 25981 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.795s user 0.028s sys 0.087s
W20260807 11:10:24.446009 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 1.79639e+06us
I20260807 11:10:24.378808 27960 raft_consensus.cc:482] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:24.449558 27960 raft_consensus.cc:504] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:24.452380 27960 leader_election.cc:288] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:24.379896 28078 raft_consensus.cc:482] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:24.381142 27882 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.662s user 0.029s sys 0.055s
I20260807 11:10:24.382254 28175 leader_election.cc:288] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:24.382521 28205 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
W20260807 11:10:24.383288 26976 consensus_meta.cc:287] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.983s user 0.009s sys 0.005s
W20260807 11:10:24.364286 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 1.75277e+06us
W20260807 11:10:24.273041 28244 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.797s user 0.014s sys 0.021s
I20260807 11:10:24.387962 28385 raft_consensus.cc:482] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:24.357453 25921 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.699s user 0.012s sys 0.060s
I20260807 11:10:24.371109 25920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1413) took 1692 ms. Trace:
I20260807 11:10:24.304708 28386 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:24.408134 25872 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:24.416236 28152 leader_election.cc:288] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:24.473366 28498 consensus_queue.cc:1035] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 11:10:24.474278 28152 consensus_queue.cc:1035] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260807 11:10:24.420933 26033 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.849s user 0.007s sys 0.009s
I20260807 11:10:24.315255 28449 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:24.477648 26033 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.933s user 0.008s sys 0.009s
I20260807 11:10:24.423542 28079 leader_election.cc:288] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:24.481112 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 89597.2us
W20260807 11:10:24.432976 25918 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.536s user 0.003s sys 0.012s
I20260807 11:10:24.312655 28293 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:24.311514 28297 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.825s user 0.009s sys 0.063s
W20260807 11:10:24.265408 27817 consensus_meta.cc:287] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.688s user 0.004s sys 0.008s
I20260807 11:10:24.395197 25801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1457) took 1193 ms. Trace:
I20260807 11:10:24.471995 28354 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:24.472952 28252 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:24.497398 25801 rpcz_store.cc:276] 0807 11:10:23.175440 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:23.176438 (+ 998us) service_pool.cc:227] Handling call
0807 11:10:23.176702 (+ 264us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:23.176900 (+ 198us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 11:10:23.200539 (+ 23639us) log.cc:842] Serialized 14 byte log entry
0807 11:10:24.367867 (+1167328us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:24.368024 (+ 157us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:24.368217 (+ 193us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:24.368418 (+ 201us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:24.368717 (+ 299us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":14274,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":255,"thread_start_us":1166522,"threads_started":1}
I20260807 11:10:24.499868 28354 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:24.500169 28252 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:24.429852 28143 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:24.464610 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 73706.2us
I20260807 11:10:24.503202 28354 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:24.481963 27882 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.772s user 0.029s sys 0.056s
I20260807 11:10:24.488269 28449 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:24.494266 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:24.465343 25920 rpcz_store.cc:276] 0807 11:10:22.678418 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:22.678530 (+ 112us) service_pool.cc:227] Handling call
0807 11:10:22.719738 (+ 41208us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:22.719965 (+ 227us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 11:10:23.881104 (+1161139us) log.cc:842] Serialized 14 byte log entry
0807 11:10:24.370337 (+489233us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:24.370452 (+ 115us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:24.370556 (+ 104us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:24.370686 (+ 130us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:24.370870 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1285197,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":54,"raft.queue_time_us":38158,"thread_start_us":1636091,"threads_started":2}
W20260807 11:10:24.495687 25921 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.838s user 0.012s sys 0.061s
I20260807 11:10:24.484048 28078 raft_consensus.cc:504] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:24.497836 25918 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.638s user 0.004s sys 0.012s
I20260807 11:10:24.488361 26036 raft_consensus.cc:1226] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:24.503216 28252 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:24.514461 26936 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:24.515311 28106 raft_consensus.cc:2781] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260807 11:10:24.515962 28244 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.991s user 0.014s sys 0.021s
I20260807 11:10:24.516884 28061 consensus_queue.cc:1046] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 11:10:24.517206 28205 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:24.518587 28143 consensus_queue.cc:1035] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:24.519620 28385 raft_consensus.cc:504] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:24.521715 28175 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:24.529650 26936 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:24.529958 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:10:24.531240 28242 leader_election.cc:288] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:24.536777 28503 raft_consensus.cc:2781] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:24.537436 28252 raft_consensus.cc:2781] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:24.550760 28175 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:24.551760 28252 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:24.552279 28252 raft_consensus.cc:3037] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:24.554446 28175 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:24.542910 28423 raft_consensus.cc:3032] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:24.545354 28205 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:24.535665 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 167871us
I20260807 11:10:24.527359 28106 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:24.559386 28423 raft_consensus.cc:729] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:24.584496 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:24.585809 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:24.596019 26980 consensus_meta.cc:287] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.997s user 0.004s sys 0.011s
W20260807 11:10:24.596259 25806 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.017s sys 0.025s
W20260807 11:10:24.596985 25806 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s user 0.017s sys 0.025s
I20260807 11:10:24.600809 28423 consensus_queue.cc:260] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:24.602783 28423 raft_consensus.cc:3037] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
W20260807 11:10:24.605082 28495 log.cc:924] Time spent T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.200s user 0.000s sys 0.001s
W20260807 11:10:24.606338 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 81135.6us
I20260807 11:10:24.606796 25797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 151) took 2145 ms. Trace:
I20260807 11:10:24.607215 25797 rpcz_store.cc:276] 0807 11:10:22.461621 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:22.461810 (+ 189us) service_pool.cc:227] Handling call
0807 11:10:22.480733 (+ 18923us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:22.494282 (+ 13549us) raft_consensus.cc:1520] Early marking committed up to index 5
0807 11:10:22.708737 (+214455us) log.cc:842] Serialized 14 byte log entry
0807 11:10:24.359296 (+1650559us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:24.386542 (+ 27246us) log.cc:842] Serialized 26 byte log entry
0807 11:10:24.405945 (+ 19403us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:24.406006 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:24.408712 (+ 2706us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:24.606541 (+197829us) raft_consensus.cc:1671] finished
0807 11:10:24.606561 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:24.606723 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":259874,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":22371,"raft.run_cpu_time_us":470,"raft.run_wall_time_us":4902,"thread_start_us":1826544,"threads_started":2,"wal-append.queue_time_us":1576450}
I20260807 11:10:24.611033 25981 leader_election.cc:302] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:24.612108 26035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1400) took 1637 ms. Trace:
I20260807 11:10:24.611404 28078 leader_election.cc:288] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:24.615016 28498 consensus_queue.cc:1035] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 11:10:24.620240 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1387) took 2010 ms. Trace:
W20260807 11:10:24.620934 28297 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.093s user 0.009s sys 0.064s
I20260807 11:10:24.620783 26042 rpcz_store.cc:276] 0807 11:10:22.609327 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:22.611957 (+ 2630us) service_pool.cc:227] Handling call
0807 11:10:22.650716 (+ 38759us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:22.651064 (+ 348us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:22.651100 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:22.651589 (+ 489us) log.cc:842] Serialized 26 byte log entry
0807 11:10:24.371926 (+1720337us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:10:24.372033 (+ 107us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:24.372149 (+ 116us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:24.619923 (+247774us) raft_consensus.cc:1671] finished
0807 11:10:24.619950 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:24.620142 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":98,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":24236,"thread_start_us":1661525,"threads_started":1,"wal-append.queue_time_us":1951153}
I20260807 11:10:24.623487 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:24.623610 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1420) took 1160 ms. Trace:
I20260807 11:10:24.624060 26046 rpcz_store.cc:276] 0807 11:10:23.462885 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:23.463031 (+ 146us) service_pool.cc:227] Handling call
0807 11:10:23.463440 (+ 409us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:23.463662 (+ 222us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:23.463687 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:23.492792 (+ 29105us) spinlock_profiling.cc:243] Waited 24 ms on lock 0x7b2c00107d68. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbe0a7e11 00007f3fbe0b7442 00007f3fbf6ffc11 00007f3fbf6c458a 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772
0807 11:10:23.493274 (+ 482us) log.cc:842] Serialized 26 byte log entry
0807 11:10:24.373994 (+880720us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:10:24.374124 (+ 130us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:24.374258 (+ 134us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:24.623217 (+248959us) raft_consensus.cc:1671] finished
0807 11:10:24.623241 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:24.623550 (+ 309us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":55240064,"thread_start_us":880043,"threads_started":1,"wal-append.queue_time_us":1126863}
I20260807 11:10:24.626575 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:24.626864 28385 leader_election.cc:288] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:24.627050 25798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 156) took 2034 ms. Trace:
I20260807 11:10:24.627807 28385 raft_consensus.cc:2781] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:24.627974 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:24.628384 28385 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:24.612586 26035 rpcz_store.cc:276] 0807 11:10:22.974592 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:22.974912 (+ 320us) service_pool.cc:227] Handling call
0807 11:10:22.975504 (+ 592us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:22.975956 (+ 452us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:22.975989 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:22.976563 (+ 574us) log.cc:842] Serialized 26 byte log entry
0807 11:10:24.373009 (+1396446us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:10:24.373131 (+ 122us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:24.373260 (+ 129us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:24.611738 (+238478us) raft_consensus.cc:1671] finished
0807 11:10:24.611761 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:24.612010 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1333,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":15706,"thread_start_us":1395651,"threads_started":1,"wal-append.queue_time_us":1632266}
I20260807 11:10:24.566856 28106 raft_consensus.cc:3037] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
W20260807 11:10:24.634462 26034 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.502s user 0.019s sys 0.020s
I20260807 11:10:24.628875 26046 raft_consensus.cc:3037] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:24.628703 25918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 141) took 2374 ms. Trace:
I20260807 11:10:24.636425 28385 raft_consensus.cc:3037] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
W20260807 11:10:24.664371 25986 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.012s sys 0.028s
I20260807 11:10:24.665122 28286 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:24.665546 28500 raft_consensus.cc:2781] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:24.665794 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:24.666190 28500 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:24.666637 28500 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:24.666738 26034 raft_consensus.cc:2454] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:24.637508 25869 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:24.631487 28503 raft_consensus.cc:686] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:24.585992 28449 consensus_queue.cc:1046] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.056s
W20260807 11:10:24.665110 25986 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.549s user 0.012s sys 0.028s
I20260807 11:10:24.669137 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:24.616425 28078 consensus_queue.cc:1035] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260807 11:10:24.677127 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:24.679056 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:24.681151 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 68578.7us
I20260807 11:10:24.640487 25918 rpcz_store.cc:276] 0807 11:10:22.254140 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:22.254666 (+ 526us) service_pool.cc:227] Handling call
0807 11:10:22.255000 (+ 334us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:22.259057 (+ 4057us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:22.259099 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:22.855921 (+596822us) spinlock_profiling.cc:243] Waited 3.94 ms on lock 0x7b2c0004b3f8. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbe0a7e11 00007f3fbe0b7442 00007f3fbf6ffc11 00007f3fbf6c458a 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772
0807 11:10:22.856394 (+ 473us) log.cc:842] Serialized 26 byte log entry
0807 11:10:24.599265 (+1742871us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:10:24.599396 (+ 131us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:24.599542 (+ 146us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:24.628303 (+ 28761us) raft_consensus.cc:1671] finished
0807 11:10:24.628322 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:24.628619 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3623,"raft.queue_time_us":93225,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":7755,"spinlock_wait_cycles":9064192,"thread_start_us":1649656,"threads_started":1,"wal-append.queue_time_us":1763794}
I20260807 11:10:24.692688 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:24.698339 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:24.702543 26036 raft_consensus.cc:3037] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:24.703111 28293 consensus_queue.cc:1046] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:24.704957 28293 raft_consensus.cc:3032] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:24.715234 26037 raft_consensus.cc:2396] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 11:10:24.705431 28293 raft_consensus.cc:729] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:24.714267 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1410) took 1534 ms. Trace:
I20260807 11:10:24.722208 28533 raft_consensus.cc:2726] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260807 11:10:24.722805 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 197758us
I20260807 11:10:24.723278 28539 raft_consensus.cc:482] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:24.725554 25872 leader_election.cc:302] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:10:24.726367 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 196994us
I20260807 11:10:24.727392 26936 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:24.729238 28503 consensus_queue.cc:237] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:24.729249 28425 raft_consensus.cc:2781] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:24.733119 28538 raft_consensus.cc:482] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:24.733505 28425 raft_consensus.cc:686] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:24.733970 28538 raft_consensus.cc:504] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:24.734784 28425 consensus_queue.cc:237] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:24.735620 26034 raft_consensus.cc:1226] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20260807 11:10:24.737689 28526 log.cc:924] Time spent T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.223s user 0.004s sys 0.000s
I20260807 11:10:24.741317 28488 raft_consensus.cc:2726] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:24.627486 25798 rpcz_store.cc:276] 0807 11:10:22.592727 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:22.783180 (+190453us) service_pool.cc:227] Handling call
0807 11:10:22.783591 (+ 411us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:22.783979 (+ 388us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:22.784014 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:22.784521 (+ 507us) log.cc:842] Serialized 26 byte log entry
0807 11:10:24.362767 (+1578246us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:10:24.362946 (+ 179us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:24.363171 (+ 225us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:24.626813 (+263642us) raft_consensus.cc:1671] finished
0807 11:10:24.626833 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:24.626995 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":36992,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":15896,"thread_start_us":1577437,"threads_started":1,"wal-append.queue_time_us":1839835}
I20260807 11:10:24.736505 28538 leader_election.cc:288] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:10:24.776353 28540 log.cc:924] Time spent T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.055s user 0.000s sys 0.001s
I20260807 11:10:24.776199 28533 raft_consensus.cc:482] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:24.776973 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:24.784507 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 60679.8us
I20260807 11:10:24.782684 25918 raft_consensus.cc:2454] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 1.
I20260807 11:10:24.721596 26039 rpcz_store.cc:276] 0807 11:10:23.176405 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:23.177471 (+ 1066us) service_pool.cc:227] Handling call
0807 11:10:23.198970 (+ 21499us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:23.199145 (+ 175us) raft_consensus.cc:1520] Early marking committed up to index 2
0807 11:10:24.369888 (+1170743us) log.cc:842] Serialized 14 byte log entry
0807 11:10:24.682547 (+312659us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:24.688944 (+ 6397us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:24.700982 (+ 12038us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:24.708885 (+ 7903us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:24.711218 (+ 2333us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2098,"prepare.queue_time_us":1412865,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":65,"raft.queue_time_us":5521,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":12163,"thread_start_us":1432344,"threads_started":2,"wal-append.queue_time_us":336833}
I20260807 11:10:24.796496 26936 leader_election.cc:302] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:24.810175 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:24.814810 28192 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:24.814672 28244 consensus_queue.cc:1046] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:24.815665 28192 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:24.817287 25752 leader_election.cc:302] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:24.818293 28192 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:24.818910 28192 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:24.819559 28242 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:24.791828 28538 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:24.821197 28246 raft_consensus.cc:2781] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:24.824201 28246 raft_consensus.cc:482] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:24.833200 28246 raft_consensus.cc:3037] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
W20260807 11:10:24.840994 28543 log.cc:924] Time spent T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.059s user 0.000s sys 0.001s
I20260807 11:10:24.820418 28242 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:24.787340 28078 consensus_queue.cc:1046] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20260807 11:10:24.843964 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:24.819533 28192 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:24.844534 28297 raft_consensus.cc:2726] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:24.812305 28498 raft_consensus.cc:482] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:24.821223 28538 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:24.864878 28143 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:24.791184 28539 raft_consensus.cc:504] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:24.833916 26976 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50942 (request call id 311) took 1625 ms. Trace:
I20260807 11:10:24.824513 28533 raft_consensus.cc:504] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:24.869207 28242 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:24.863314 26042 raft_consensus.cc:2454] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
W20260807 11:10:24.883814 28537 log.cc:924] Time spent T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20260807 11:10:24.884156 25869 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:24.885075 28498 raft_consensus.cc:504] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:24.888017 28498 leader_election.cc:288] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:24.888067 27882 raft_consensus.cc:2726] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:24.884995 28489 raft_consensus.cc:3032] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Stepping down as leader of term 3
W20260807 11:10:24.874881 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 161901us
W20260807 11:10:24.889487 28487 log.cc:924] Time spent T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.072s user 0.000s sys 0.001s
W20260807 11:10:24.872928 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 118645us
I20260807 11:10:24.888938 28192 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:24.876984 26039 raft_consensus.cc:2354] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4.
I20260807 11:10:24.897189 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1416) took 1529 ms. Trace:
I20260807 11:10:24.876610 26976 rpcz_store.cc:276] 0807 11:10:23.186517 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:23.189010 (+ 2493us) service_pool.cc:227] Handling call
0807 11:10:23.189452 (+ 442us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:23.189809 (+ 357us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 11:10:24.516385 (+1326576us) log.cc:842] Serialized 14 byte log entry
0807 11:10:24.719791 (+203406us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:24.740067 (+ 20276us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:24.760985 (+ 20918us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:24.778949 (+ 17964us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:24.806259 (+ 27310us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4374,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":2305,"thread_start_us":183151,"threads_started":1,"wal-append.queue_time_us":331132}
I20260807 11:10:24.864265 28293 consensus_queue.cc:260] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:24.916339 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:24.913780 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 148) took 2308 ms. Trace:
I20260807 11:10:24.891384 28489 raft_consensus.cc:729] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
W20260807 11:10:24.925755 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 56263.4us
I20260807 11:10:24.924674 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:24.929003 26042 raft_consensus.cc:2454] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:24.918948 25921 rpcz_store.cc:276] 0807 11:10:22.605572 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:22.605708 (+ 136us) service_pool.cc:227] Handling call
0807 11:10:22.606004 (+ 296us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:22.606308 (+ 304us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:22.606340 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:22.606757 (+ 417us) log.cc:842] Serialized 26 byte log entry
0807 11:10:24.512270 (+1905513us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:10:24.512412 (+ 142us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:24.512556 (+ 144us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:24.891306 (+378750us) raft_consensus.cc:1671] finished
0807 11:10:24.891332 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:24.913705 (+ 22373us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13452,"raft.queue_time_us":7458,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":8682,"thread_start_us":1850947,"threads_started":1,"wal-append.queue_time_us":2148638}
I20260807 11:10:24.936060 28533 leader_election.cc:288] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:24.937381 28252 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:24.942576 25806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 153) took 2347 ms. Trace:
I20260807 11:10:24.943578 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:24.948437 26046 raft_consensus.cc:2454] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:24.946610 25750 leader_election.cc:302] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:24.951756 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:24.954562 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 63825us
I20260807 11:10:24.956398 25752 leader_election.cc:302] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:24.951326 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:24.960907 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 74187.4us
I20260807 11:10:24.962003 28456 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:24.962642 28456 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:24.963909 28462 consensus_queue.cc:1046] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.084s
I20260807 11:10:24.966619 28252 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:24.924116 28143 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:24.970309 28143 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:24.971585 26039 raft_consensus.cc:2454] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:24.974772 28286 consensus_queue.cc:1035] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:24.977425 28246 raft_consensus.cc:504] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:24.977774 25750 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:24.977918 27817 consensus_queue.cc:1035] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:24.979990 25750 leader_election.cc:302] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:24.982134 28489 consensus_queue.cc:260] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:24.983551 28489 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:24.985422 28242 raft_consensus.cc:2726] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:24.986872 28242 raft_consensus.cc:2726] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:24.987632 28242 raft_consensus.cc:2781] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:24.988029 28242 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:24.988425 28242 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:24.949908 28106 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:24.989449 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:24.990728 26976 raft_consensus.cc:2396] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260807 11:10:24.990887 28106 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:24.992210 28538 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:24.898370 26033 rpcz_store.cc:276] 0807 11:10:23.367992 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:23.368116 (+ 124us) service_pool.cc:227] Handling call
0807 11:10:23.385393 (+ 17277us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:23.385889 (+ 496us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:23.385933 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:23.497430 (+111497us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b2c00107d68. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbe0a7e11 00007f3fbe0b7442 00007f3fbf6ffc11 00007f3fbf6c458a 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772
0807 11:10:23.497821 (+ 391us) log.cc:842] Serialized 26 byte log entry
0807 11:10:24.478466 (+980645us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:24.850528 (+372062us) log.cc:842] Serialized 14 byte log entry
0807 11:10:24.855429 (+ 4901us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:24.859318 (+ 3889us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:24.859363 (+ 45us) raft_consensus.cc:1671] finished
0807 11:10:24.873570 (+ 14207us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:24.897116 (+ 23546us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":72643,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":74,"raft.queue_time_us":47525,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":289,"spinlock_wait_cycles":234685184,"thread_start_us":1184563,"threads_started":2,"wal-append.queue_time_us":1189320}
W20260807 11:10:24.993542 28554 log.cc:924] Time spent T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.113s user 0.001s sys 0.000s
I20260807 11:10:24.947371 25806 rpcz_store.cc:276] 0807 11:10:22.591642 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:22.599463 (+ 7821us) service_pool.cc:227] Handling call
0807 11:10:22.599802 (+ 339us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:23.883023 (+1283221us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:23.883068 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:23.883573 (+ 505us) log.cc:842] Serialized 26 byte log entry
0807 11:10:24.597777 (+714204us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:10:24.597896 (+ 119us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:24.598067 (+ 171us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:24.926504 (+328437us) raft_consensus.cc:1671] finished
0807 11:10:24.926533 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:24.938636 (+ 12103us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14362,"raft.queue_time_us":136448,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":9850,"thread_start_us":1851522,"threads_started":2,"wal-append.queue_time_us":729127}
W20260807 11:10:24.994676 28557 log.cc:924] Time spent T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
I20260807 11:10:24.994971 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:24.995888 26976 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:24.996178 28483 raft_consensus.cc:3032] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:24.996615 28483 raft_consensus.cc:729] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:24.997645 28483 consensus_queue.cc:260] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:24.998870 28483 raft_consensus.cc:3037] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:24.999573 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:24.972708 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:25.010928 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 56576.9us
I20260807 11:10:25.012106 25752 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:25.013327 25919 raft_consensus.cc:1226] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:25.013438 28561 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:25.014181 28561 raft_consensus.cc:504] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:25.016696 28561 leader_election.cc:288] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:25.017082 28293 raft_consensus.cc:3037] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:24.977872 25869 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:25.019569 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:24.941002 28192 consensus_queue.cc:1035] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.070s
I20260807 11:10:25.033934 26039 raft_consensus.cc:2454] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:24.979990 28246 leader_election.cc:288] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:25.040896 25872 leader_election.cc:398] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Vote denied by peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009) with higher term. Message: Invalid argument: T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4.
I20260807 11:10:25.052112 26037 raft_consensus.cc:2454] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:25.043525 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.044320 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.045320 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.008350 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:25.022719 28498 raft_consensus.cc:2781] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:25.052197 25872 leader_election.cc:401] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260807 11:10:25.041057 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.011210 26980 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50942 (request call id 312) took 1822 ms. Trace:
I20260807 11:10:25.046936 26976 raft_consensus.cc:2454] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:25.049536 25752 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:25.057300 25917 raft_consensus.cc:2379] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 3: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:25.045364 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:24.956326 28500 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:25.054742 28538 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:25.071557 28538 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:25.071936 26936 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:25.076876 27960 raft_consensus.cc:2781] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:25.077575 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:25.078284 28061 raft_consensus.cc:2781] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:25.078403 25806 raft_consensus.cc:3037] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:25.078807 28061 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:25.079260 28061 raft_consensus.cc:3037] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:25.080663 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:25.081389 25800 raft_consensus.cc:3037] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:25.077419 27960 raft_consensus.cc:686] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:25.081872 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:25.082880 27960 consensus_queue.cc:237] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:25.053179 28561 raft_consensus.cc:482] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:25.081382 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:25.084367 28561 raft_consensus.cc:504] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:25.085269 25801 raft_consensus.cc:2454] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:25.059386 26980 rpcz_store.cc:276] 0807 11:10:23.186876 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:23.200264 (+ 13388us) service_pool.cc:227] Handling call
0807 11:10:23.212701 (+ 12437us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:23.216911 (+ 4210us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 11:10:24.596931 (+1380020us) log.cc:842] Serialized 14 byte log entry
0807 11:10:24.938813 (+341882us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:24.957922 (+ 19109us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:24.958013 (+ 91us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:24.958120 (+ 107us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:25.009328 (+ 51208us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10580,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":129,"thread_start_us":320405,"threads_started":2,"wal-append.queue_time_us":273106}
I20260807 11:10:25.060612 25932 raft_consensus.cc:2454] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:25.094779 25920 raft_consensus.cc:2396] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 11:10:25.068359 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:25.069610 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 91864.9us
I20260807 11:10:25.059993 28498 raft_consensus.cc:482] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:25.069273 28246 raft_consensus.cc:2781] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:25.111114 25806 raft_consensus.cc:2454] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:25.116336 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:25.117291 26046 raft_consensus.cc:3037] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 4
W20260807 11:10:25.115301 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 156799us
I20260807 11:10:25.118597 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.115513 25752 leader_election.cc:302] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:25.119613 25918 raft_consensus.cc:2454] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:25.118248 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.136289 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.113668 28498 raft_consensus.cc:3037] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
W20260807 11:10:25.138856 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 65156.6us
I20260807 11:10:25.097622 25925 raft_consensus.cc:2454] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:25.142050 25872 leader_election.cc:302] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:25.142620 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:25.144305 25800 raft_consensus.cc:2454] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:25.114430 28246 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:25.149022 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.139746 28079 raft_consensus.cc:2781] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:25.156479 28539 leader_election.cc:288] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:10:25.161069 28524 log.cc:924] Time spent T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.491s user 0.002s sys 0.001s
I20260807 11:10:25.163234 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:25.164505 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:25.165341 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.142438 28462 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
W20260807 11:10:25.167704 28438 consensus_meta.cc:287] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.895s user 0.014s sys 0.000s
I20260807 11:10:25.167845 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:25.168845 25801 raft_consensus.cc:2379] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 2: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:25.169220 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:25.169775 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:25.170058 25799 raft_consensus.cc:2454] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:25.170511 25797 raft_consensus.cc:2454] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:25.171573 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:25.172899 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:25.173698 25800 raft_consensus.cc:2354] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4.
I20260807 11:10:25.173733 28088 raft_consensus.cc:2781] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:25.174302 28088 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:25.171285 28385 raft_consensus.cc:504] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:25.177943 25920 raft_consensus.cc:3037] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:25.174738 28088 raft_consensus.cc:3037] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:25.169776 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:25.154070 28246 raft_consensus.cc:3037] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:25.185606 26033 raft_consensus.cc:1226] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260807 11:10:25.187493 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:25.189136 28511 raft_consensus.cc:3032] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:25.189610 28511 raft_consensus.cc:729] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:25.190584 28511 consensus_queue.cc:260] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:25.191406 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.150202 28566 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:25.192873 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:25.155859 28079 raft_consensus.cc:482] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:25.200198 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:25.200615 26035 raft_consensus.cc:2379] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 3: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:25.198077 26036 raft_consensus.cc:2454] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:25.168192 28438 raft_consensus.cc:504] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:25.192000 28511 raft_consensus.cc:3037] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:25.173875 28538 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:25.211123 25921 raft_consensus.cc:2396] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 11:10:25.212776 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:25.212985 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:25.216073 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.217123 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:25.217833 25797 raft_consensus.cc:2454] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:10:25.227542 28500 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:25.227653 25799 raft_consensus.cc:2454] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:25.227896 28566 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:25.202900 28079 raft_consensus.cc:3037] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
W20260807 11:10:25.235319 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 94932.4us
I20260807 11:10:25.228461 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:25.241770 25752 leader_election.cc:302] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:25.245440 25806 raft_consensus.cc:2396] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260807 11:10:25.238754 25796 raft_consensus.cc:2454] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:10:25.249224 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:25.254026 25872 leader_election.cc:302] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:25.193516 26976 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:25.268222 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.240835 28106 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:25.274887 26046 raft_consensus.cc:2454] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:25.277797 25799 raft_consensus.cc:2454] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:25.296490 25926 raft_consensus.cc:2454] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:25.298929 25752 leader_election.cc:302] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:25.266160 28457 raft_consensus.cc:2781] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:25.303277 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:25.304917 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 150460us
I20260807 11:10:25.308002 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.272781 28425 raft_consensus.cc:2781] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:25.308857 25921 raft_consensus.cc:2454] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
W20260807 11:10:25.300714 28503 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.018s sys 0.025s
I20260807 11:10:25.279263 28293 raft_consensus.cc:2781] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:25.314143 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:25.282919 28538 raft_consensus.cc:482] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:25.325135 25796 raft_consensus.cc:2454] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:25.318761 26936 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:25.311553 28106 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:25.334656 28106 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:25.312273 28425 raft_consensus.cc:482] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:25.336997 28425 raft_consensus.cc:3037] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:25.340862 25799 raft_consensus.cc:2379] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 3: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
W20260807 11:10:25.315817 28503 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.536s user 0.018s sys 0.025s
I20260807 11:10:25.341307 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.342109 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:25.342568 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.342847 26976 raft_consensus.cc:2454] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:25.343724 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.343956 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.344308 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:25.344828 25916 raft_consensus.cc:1226] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:25.342568 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:25.345745 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.346033 26980 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:25.347131 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:25.347949 26976 raft_consensus.cc:2354] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4.
I20260807 11:10:25.348570 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.345221 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.349234 28570 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:25.350234 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:10:25.350824 28545 log.cc:924] Time spent T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.536s user 0.003s sys 0.000s
I20260807 11:10:25.350608 28570 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:25.353441 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:25.353981 28293 raft_consensus.cc:686] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:25.354961 28570 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:25.355096 28293 consensus_queue.cc:237] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:25.355826 28252 raft_consensus.cc:482] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:25.356283 28252 raft_consensus.cc:504] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:25.357296 28385 leader_election.cc:288] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:25.358166 28252 leader_election.cc:288] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:25.303663 28457 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:25.296336 27960 raft_consensus.cc:482] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:25.367287 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:25.370429 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 119750us
W20260807 11:10:25.371969 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 53329.1us
I20260807 11:10:25.372372 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:25.314103 28483 raft_consensus.cc:2781] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:25.376011 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.330883 28205 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:25.378579 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:25.378657 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.305219 28566 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:25.353469 28438 leader_election.cc:288] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:25.386121 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.367125 28457 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:25.383152 25921 raft_consensus.cc:2454] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:25.390520 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.366957 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.376294 28462 raft_consensus.cc:2726] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009) with higher term. Message: Invalid argument: T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4.
I20260807 11:10:25.397615 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:10:25.398137 28521 log.cc:924] Time spent T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20260807 11:10:25.398280 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:25.375579 28538 raft_consensus.cc:504] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:25.399387 28079 raft_consensus.cc:504] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:25.417555 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.418242 25926 raft_consensus.cc:2454] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:25.418550 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.401026 28577 raft_consensus.cc:2726] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:25.392588 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:25.401824 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:25.423141 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:25.401990 28242 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:25.369199 28061 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:25.403599 28246 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:25.406076 25750 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:25.382931 28385 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:25.409874 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.416296 26041 raft_consensus.cc:2379] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 4: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
W20260807 11:10:25.420257 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 118568us
I20260807 11:10:25.402299 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.426092 26987 raft_consensus.cc:2396] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260807 11:10:25.407738 26980 raft_consensus.cc:1226] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260807 11:10:25.423800 25799 raft_consensus.cc:2454] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:25.447953 26976 raft_consensus.cc:2379] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 2: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:25.449450 28246 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:25.447306 28061 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195)
I20260807 11:10:25.393075 28498 raft_consensus.cc:504] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:25.388600 28205 raft_consensus.cc:504] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:25.395588 27960 raft_consensus.cc:504] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:25.445114 28386 raft_consensus.cc:482] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:25.489221 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:25.445729 28594 log.cc:924] Time spent T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260807 11:10:25.474875 25932 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:25.391748 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:25.422549 28483 raft_consensus.cc:686] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:25.450712 25920 raft_consensus.cc:2454] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
W20260807 11:10:25.485222 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 79245.9us
I20260807 11:10:25.492170 25931 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:10:25.492668 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:25.508711 25917 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:25.493615 27960 leader_election.cc:288] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:25.494095 28386 raft_consensus.cc:504] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:25.495806 28079 leader_election.cc:288] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:25.495916 28242 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:25.496490 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:25.496693 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:25.497498 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:25.421695 28088 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:25.499054 28425 raft_consensus.cc:504] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:25.499634 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:25.500450 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:25.501435 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:25.500451 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:25.504710 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:25.490464 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:25.535275 26034 raft_consensus.cc:3037] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:25.535404 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:25.536053 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:25.537168 26978 raft_consensus.cc:3037] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:25.537448 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:25.537673 28386 leader_election.cc:288] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:25.538336 26979 raft_consensus.cc:2379] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 4: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:25.540520 28533 raft_consensus.cc:3032] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:25.540935 28533 raft_consensus.cc:729] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:25.541807 28533 consensus_queue.cc:260] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:25.537179 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:25.506476 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:25.508312 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:25.511461 28175 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:25.511862 28078 raft_consensus.cc:2781] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:25.522171 25926 raft_consensus.cc:2379] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 3: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:25.523927 26041 raft_consensus.cc:2379] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 3: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:25.498822 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:25.501869 28498 leader_election.cc:288] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:25.484491 28385 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:25.533645 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:25.552209 25797 raft_consensus.cc:2379] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 2: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:25.503873 28603 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:25.618815 25919 raft_consensus.cc:2354] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3.
I20260807 11:10:25.618914 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:25.602648 26033 raft_consensus.cc:2454] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:25.620281 25800 raft_consensus.cc:2454] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:25.632362 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:25.639302 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:25.551724 28078 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:25.667629 26978 raft_consensus.cc:2454] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:25.682936 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:25.684270 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:25.684944 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:25.701277 28088 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:25.701131 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:25.703003 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:25.699132 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:25.715139 26041 raft_consensus.cc:3037] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
W20260807 11:10:25.720711 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 46627.8us
I20260807 11:10:25.724076 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:25.726096 28538 leader_election.cc:288] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:25.684964 28425 leader_election.cc:288] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:25.726048 26974 raft_consensus.cc:2354] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4.
I20260807 11:10:25.729311 26980 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
W20260807 11:10:25.729782 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 43897.3us
I20260807 11:10:25.731247 28385 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:25.734671 25872 leader_election.cc:302] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:25.662299 28078 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:25.752542 28567 consensus_queue.cc:1046] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 11:10:25.651130 28175 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:25.753816 28567 raft_consensus.cc:3032] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:25.833794 28561 leader_election.cc:288] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:25.836668 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:10:25.840946 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 106342us
I20260807 11:10:25.842680 26038 raft_consensus.cc:1261] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260807 11:10:25.829604 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 82264.7us
I20260807 11:10:25.833947 26035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 1507) took 1219 ms. Trace:
I20260807 11:10:25.846530 26035 rpcz_store.cc:276] 0807 11:10:24.614922 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:24.659569 (+ 44647us) service_pool.cc:227] Handling call
0807 11:10:25.833714 (+1174145us) spinlock_profiling.cc:243] Waited 505 ms on lock 0x7b5400229a54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:25.833858 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1162092288}
I20260807 11:10:25.686448 28603 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:25.760857 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:25.668903 28483 consensus_queue.cc:237] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:25.857064 26042 raft_consensus.cc:2454] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:25.863730 25981 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:25.831123 28567 raft_consensus.cc:729] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:25.875979 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:25.876577 26038 raft_consensus.cc:3037] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 3
W20260807 11:10:26.019688 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 164703us
I20260807 11:10:25.878100 28244 consensus_queue.cc:1035] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.115s
I20260807 11:10:25.883448 26978 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:10:26.031515 28143 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:26.032132 28293 raft_consensus.cc:482] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:25.859072 28286 raft_consensus.cc:2781] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:25.903023 28457 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:26.041333 26041 raft_consensus.cc:2454] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:26.043834 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:26.061334 25752 leader_election.cc:302] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:26.061386 25799 raft_consensus.cc:2454] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
W20260807 11:10:26.045894 28579 log.cc:924] Time spent T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.310s user 0.000s sys 0.010s
W20260807 11:10:26.049558 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 142482us
W20260807 11:10:26.050748 26034 consensus_meta.cc:287] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.515s user 0.014s sys 0.000s
I20260807 11:10:26.046758 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:26.075098 26034 raft_consensus.cc:2454] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
W20260807 11:10:26.057438 25869 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.006s sys 0.013s
W20260807 11:10:26.076134 25869 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.566s user 0.006s sys 0.014s
I20260807 11:10:26.076196 26039 raft_consensus.cc:1226] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:26.046608 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
W20260807 11:10:26.054314 28545 log.cc:924] Time spent T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.170s user 0.000s sys 0.005s
I20260807 11:10:26.065148 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:26.077680 25981 leader_election.cc:302] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:26.070698 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:26.076833 26034 raft_consensus.cc:1261] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:10:26.075309 25786 raft_consensus.cc:2454] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:26.101378 25752 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:26.059149 28286 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:26.077834 28143 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:26.103511 28538 raft_consensus.cc:2781] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:26.105662 25869 leader_election.cc:302] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:26.108876 28293 raft_consensus.cc:504] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:26.110110 25918 raft_consensus.cc:2454] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:26.072939 28498 raft_consensus.cc:2781] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election won for term 3
W20260807 11:10:26.123627 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 93849.4us
I20260807 11:10:26.125030 28457 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:26.112534 28613 log.cc:924] Time spent T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.359s user 0.000s sys 0.008s
I20260807 11:10:26.070165 28567 consensus_queue.cc:260] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:26.125929 28603 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:26.128908 28175 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:26.131181 25800 raft_consensus.cc:1261] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:10:26.136070 28078 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:26.138407 25799 raft_consensus.cc:1226] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:26.041062 28566 raft_consensus.cc:2781] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260807 11:10:26.136519 28614 log.cc:924] Time spent T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.211s user 0.004s sys 0.002s
W20260807 11:10:26.146207 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 44916.6us
I20260807 11:10:26.151669 25750 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
W20260807 11:10:26.149223 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 43697.6us
I20260807 11:10:26.064056 25797 raft_consensus.cc:2932] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:26.147650 28538 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:26.145532 28286 raft_consensus.cc:3037] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:26.165259 25869 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:26.191331 25799 raft_consensus.cc:3032] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:26.191782 25799 raft_consensus.cc:729] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:26.150656 28293 leader_election.cc:288] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:26.150820 28616 raft_consensus.cc:2726] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:26.147900 28619 raft_consensus.cc:2781] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:26.203193 28619 raft_consensus.cc:686] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 2 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:26.204125 28619 consensus_queue.cc:237] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:26.164876 28244 raft_consensus.cc:2781] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:26.206774 28244 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:26.207180 28244 raft_consensus.cc:3037] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:26.193800 28079 consensus_queue.cc:1035] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260807 11:10:26.209408 28457 raft_consensus.cc:2781] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260807 11:10:26.209777 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 58220.7us
I20260807 11:10:26.210690 25750 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:26.212069 25750 leader_election.cc:302] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:26.212308 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:26.195478 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:26.164908 28538 raft_consensus.cc:3037] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:26.150497 28498 raft_consensus.cc:686] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:26.197979 28143 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:26.222559 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 57392.9us
I20260807 11:10:26.222604 25750 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:26.209841 28457 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:26.226370 28457 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:26.226532 28386 raft_consensus.cc:2726] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:26.218181 28483 raft_consensus.cc:2726] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:26.229453 28498 consensus_queue.cc:237] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:26.231480 28293 raft_consensus.cc:2781] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:26.231967 28293 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:26.232465 28293 raft_consensus.cc:3037] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:26.210644 25799 consensus_queue.cc:260] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:26.163782 28566 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:26.218731 26038 raft_consensus.cc:2454] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:26.240839 25869 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:10:26.242329 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 130798us
I20260807 11:10:26.242919 25799 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:26.243666 25750 leader_election.cc:302] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:26.218098 28616 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:26.239732 28566 raft_consensus.cc:3037] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:26.257753 25869 leader_election.cc:398] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4.
I20260807 11:10:26.242666 28503 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:26.286520 25869 leader_election.cc:302] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:26.248131 28078 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:26.310396 25750 leader_election.cc:302] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:26.461097 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:26.469203 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:26.471995 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 368136us
I20260807 11:10:26.298223 28425 raft_consensus.cc:2726] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 11:10:26.250156 28143 raft_consensus.cc:2781] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:26.262945 28386 raft_consensus.cc:2726] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260807 11:10:26.486686 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 221925us
I20260807 11:10:26.492951 25800 raft_consensus.cc:1226] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:26.490984 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:26.488484 25981 leader_election.cc:302] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
W20260807 11:10:26.484483 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 235543us
I20260807 11:10:26.481118 25806 raft_consensus.cc:2454] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
W20260807 11:10:26.505625 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 244966us
I20260807 11:10:26.520021 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:26.520800 25796 raft_consensus.cc:3037] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:26.252455 28616 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:26.491608 28618 raft_consensus.cc:2726] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:26.490491 28079 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:26.568302 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:26.497121 28106 raft_consensus.cc:482] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:26.497299 28462 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:26.583638 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:26.503495 28425 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:26.481441 28143 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:26.629158 25800 raft_consensus.cc:2354] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for earlier term 4. Current term is 5.
W20260807 11:10:26.637348 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 160252us
W20260807 11:10:26.640072 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 138294us
I20260807 11:10:26.506243 28489 consensus_queue.cc:1046] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.080s
I20260807 11:10:26.643043 25797 raft_consensus.cc:1226] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:26.654335 25981 leader_election.cc:302] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:26.299582 28503 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:26.637144 28143 raft_consensus.cc:3037] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:26.648360 25750 leader_election.cc:302] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:26.659683 25981 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:26.644336 28631 raft_consensus.cc:2781] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:26.643667 28606 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:26.665063 28606 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:26.668027 28637 log.cc:924] Time spent T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.137s user 0.000s sys 0.000s
W20260807 11:10:26.668957 28638 log.cc:924] Time spent T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
I20260807 11:10:26.669734 28456 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:26.671268 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:26.672423 25801 raft_consensus.cc:2379] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 3: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:26.674403 28561 raft_consensus.cc:2781] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:26.625962 28106 raft_consensus.cc:504] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:26.678721 26034 raft_consensus.cc:1261] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 11:10:26.626461 28079 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:26.693198 28079 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:26.693353 28385 raft_consensus.cc:2781] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:26.690373 28538 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:26.694108 28385 raft_consensus.cc:686] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:26.631906 28462 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:26.695384 28385 consensus_queue.cc:237] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:26.696097 28462 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:26.697914 28538 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:26.648350 28627 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:26.708541 28483 consensus_queue.cc:1035] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260807 11:10:26.713140 28286 consensus_meta.cc:287] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.518s user 0.011s sys 0.000s
I20260807 11:10:26.706880 28561 raft_consensus.cc:686] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:26.713727 28286 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:26.717105 25799 raft_consensus.cc:1261] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260807 11:10:26.717597 25799 pending_rounds.cc:76] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815: Aborting all ops after (but not including) 5
I20260807 11:10:26.717885 26041 raft_consensus.cc:1261] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:10:26.717855 25799 pending_rounds.cc:97] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260807 11:10:26.722805 25799 raft_consensus.cc:2866] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:10:26.723922 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:26.727011 25981 leader_election.cc:302] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:26.724817 26034 raft_consensus.cc:2454] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:26.677695 28631 raft_consensus.cc:482] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
W20260807 11:10:26.735855 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 94869.6us
I20260807 11:10:26.728813 28462 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:26.694132 28106 leader_election.cc:288] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:26.731801 25752 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:26.652351 28425 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:26.739739 25981 leader_election.cc:302] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:26.718859 28619 raft_consensus.cc:2726] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:26.755986 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:26.679049 28244 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:26.737987 28606 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:26.744359 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:26.714367 28561 consensus_queue.cc:237] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:26.735954 28631 raft_consensus.cc:3037] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:26.807822 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:26.729928 28483 consensus_queue.cc:1035] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260807 11:10:26.739732 28626 raft_consensus.cc:2781] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:26.818661 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:26.757133 28538 raft_consensus.cc:2781] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:26.755272 28106 raft_consensus.cc:482] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:26.805550 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:26.822125 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:26.824895 28244 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:26.752825 28269 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
W20260807 11:10:26.835461 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 95826.3us
W20260807 11:10:26.769094 28293 consensus_meta.cc:287] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.506s user 0.009s sys 0.000s
I20260807 11:10:26.725679 28627 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:26.814033 28651 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:26.749604 28462 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:26.816826 26978 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
W20260807 11:10:26.853475 26976 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 318) took 1504 ms (client timeout 1557 ms). Trace:
W20260807 11:10:26.854090 26976 rpcz_store.cc:269] 0807 11:10:25.349420 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:25.349899 (+ 479us) service_pool.cc:227] Handling call
0807 11:10:26.853200 (+1503301us) spinlock_profiling.cc:243] Waited 96.8 ms on lock 0x7b540045bfd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:26.853380 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":222603520}
I20260807 11:10:26.854883 26987 raft_consensus.cc:1226] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 1. Current term is 2. Ops: []
W20260807 11:10:26.855648 28639 log.cc:924] Time spent T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.127s user 0.000s sys 0.001s
I20260807 11:10:26.845765 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:26.846695 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:26.857234 26987 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 364) took 1359 ms. Trace:
I20260807 11:10:26.857777 28425 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:26.857693 26987 rpcz_store.cc:276] 0807 11:10:25.497654 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:25.512778 (+ 15124us) service_pool.cc:227] Handling call
0807 11:10:26.854685 (+1341907us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:26.855377 (+ 692us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:26.857055 (+ 1678us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b540045bfd0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:26.857182 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3050712320}
I20260807 11:10:26.819368 28619 raft_consensus.cc:2781] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:26.849905 25981 leader_election.cc:398] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195) with higher term. Message: Invalid argument: T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4.
I20260807 11:10:26.826038 28626 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:26.826371 28538 raft_consensus.cc:482] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:26.853044 28462 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:26.848459 28269 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:26.864228 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 132516us
I20260807 11:10:26.864593 25981 leader_election.cc:401] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260807 11:10:26.828842 28616 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:26.829627 26033 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.024s sys 0.029s
I20260807 11:10:26.867234 28619 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 11:10:26.952203 26033 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.686s user 0.030s sys 0.030s
I20260807 11:10:26.956254 28619 raft_consensus.cc:3037] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:26.871421 28538 raft_consensus.cc:3037] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:27.071422 25796 raft_consensus.cc:2454] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
W20260807 11:10:26.961992 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 191649us
I20260807 11:10:26.870306 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:26.951839 28385 raft_consensus.cc:2726] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195) with higher term. Message: Invalid argument: T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4.
I20260807 11:10:26.871021 28626 raft_consensus.cc:3037] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:27.076707 25869 leader_election.cc:302] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:26.861657 28651 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:26.846239 28293 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:26.954094 28286 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:26.860706 28106 raft_consensus.cc:504] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:27.088368 28106 leader_election.cc:288] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:26.863627 28456 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:27.062006 25750 leader_election.cc:302] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:27.062686 25981 leader_election.cc:302] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:10:27.053203 26936 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.000s sys 0.011s
W20260807 11:10:27.090130 26936 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.582s user 0.000s sys 0.014s
W20260807 11:10:27.090448 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 582839us
I20260807 11:10:27.090224 28456 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:26.871803 28425 raft_consensus.cc:482] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:27.093084 28425 raft_consensus.cc:504] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:27.093593 28061 consensus_queue.cc:1035] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 11:10:27.093590 26978 raft_consensus.cc:2454] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:27.095136 28425 leader_election.cc:288] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:27.095772 26041 raft_consensus.cc:1226] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:26.964007 28647 raft_consensus.cc:482] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:26.845863 28244 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
W20260807 11:10:27.073922 28457 consensus_meta.cc:287] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.847s user 0.016s sys 0.006s
I20260807 11:10:27.085141 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:27.062285 28616 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:27.072948 28078 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:27.082321 28503 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:27.096715 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:27.097014 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:27.093092 28566 consensus_meta.cc:287] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.668s user 0.014s sys 0.001s
I20260807 11:10:27.109401 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:27.145350 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:27.146860 28566 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:27.148327 26036 raft_consensus.cc:2454] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:10:27.151015 26938 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:27.151185 26936 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:27.152323 26034 raft_consensus.cc:1226] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:27.135919 28425 raft_consensus.cc:2726] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260807 11:10:27.110829 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 48230.9us
I20260807 11:10:27.155951 25981 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:27.146118 26035 raft_consensus.cc:2454] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:27.103021 28671 raft_consensus.cc:482] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:27.102982 28653 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:27.113708 28385 raft_consensus.cc:2781] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:27.116518 28269 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:27.162955 28671 raft_consensus.cc:504] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:27.164634 28538 raft_consensus.cc:504] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:27.165350 26042 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59312 (request call id 1449) took 1671 ms (client timeout 2226 ms). Trace:
W20260807 11:10:27.165723 26042 rpcz_store.cc:269] 0807 11:10:25.494356 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:25.500194 (+ 5838us) service_pool.cc:227] Handling call
0807 11:10:27.165188 (+1664994us) spinlock_profiling.cc:243] Waited 322 ms on lock 0x7b5400392bd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:27.165294 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":740507904}
I20260807 11:10:27.166628 28538 leader_election.cc:288] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:27.142692 28647 raft_consensus.cc:504] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:27.171273 28385 raft_consensus.cc:482] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:27.171847 28385 raft_consensus.cc:3037] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:27.137310 28457 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:27.174104 28627 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:27.174427 28457 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:27.175632 28566 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:27.165277 28175 raft_consensus.cc:2781] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:27.176497 28175 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:27.176916 28175 raft_consensus.cc:3037] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:27.121640 28651 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:27.184795 25796 raft_consensus.cc:1226] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:27.126430 28244 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:27.129066 28293 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:27.135790 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 55519us
W20260807 11:10:27.103906 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 232523us
I20260807 11:10:27.189632 25750 leader_election.cc:302] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:27.191274 28646 consensus_queue.cc:1046] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 11:10:27.140554 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:27.195823 26038 raft_consensus.cc:2354] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 2. Current term is 3.
I20260807 11:10:27.107661 28462 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:27.226570 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 146844us
I20260807 11:10:27.226876 28462 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:27.227456 25869 leader_election.cc:302] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:27.225493 25872 leader_election.cc:302] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:27.165198 28671 leader_election.cc:288] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:27.157371 25752 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:27.104882 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:27.148454 28078 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:27.162667 28653 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:27.171048 28647 leader_election.cc:288] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:27.119122 28631 raft_consensus.cc:504] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:27.081602 28143 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:27.133118 28483 raft_consensus.cc:2726] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:27.138464 28616 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:27.180055 28503 raft_consensus.cc:2781] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:27.161160 28565 raft_consensus.cc:2726] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 11:10:27.245496 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 88240.6us
I20260807 11:10:27.105113 28456 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:27.233304 28293 raft_consensus.cc:2726] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260807 11:10:27.234629 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 78766.6us
I20260807 11:10:27.236583 25869 leader_election.cc:302] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:27.238699 25750 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:27.248246 25752 leader_election.cc:398] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Vote denied by peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009) with higher term. Message: Invalid argument: T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 2. Current term is 3.
I20260807 11:10:27.246284 25799 raft_consensus.cc:2454] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:10:27.257040 25752 leader_election.cc:401] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260807 11:10:27.268232 25981 leader_election.cc:302] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:27.266640 28462 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:27.269786 28631 leader_election.cc:288] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:27.270651 28631 raft_consensus.cc:2781] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:27.271039 28631 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:27.271443 28631 raft_consensus.cc:3037] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:27.271494 28653 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:27.271945 25803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53324 (request call id 1400) took 1395 ms. Trace:
I20260807 11:10:27.273406 28616 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:27.275120 28078 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:27.245150 28503 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:27.276003 28503 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:27.280974 28293 raft_consensus.cc:2781] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:10:27.281524 28293 raft_consensus.cc:686] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:27.272342 25803 rpcz_store.cc:276] 0807 11:10:25.876287 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:25.876397 (+ 110us) service_pool.cc:227] Handling call
0807 11:10:27.271093 (+1394696us) spinlock_profiling.cc:243] Waited 9.39 ms on lock 0x7b5800003c98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc461a2b0 00007f3fc46057f9 00007f3fc46052a2 00007f3fbdceb684
0807 11:10:27.271337 (+ 244us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:27.271530 (+ 193us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:27.271556 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:27.271598 (+ 42us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:27.271629 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:27.271680 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:27.271779 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21601664}
I20260807 11:10:27.282370 28293 consensus_queue.cc:237] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:27.283684 28078 raft_consensus.cc:3037] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:27.283516 28175 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:27.285601 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:27.286077 28619 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:27.286609 26980 raft_consensus.cc:1261] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 11:10:27.288002 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1419) took 2099 ms. Trace:
I20260807 11:10:27.288113 28619 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:27.269778 25981 leader_election.cc:302] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:27.247681 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:27.288415 25917 rpcz_store.cc:276] 0807 11:10:25.188778 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:25.191831 (+ 3053us) service_pool.cc:227] Handling call
0807 11:10:27.287824 (+2095993us) spinlock_profiling.cc:243] Waited 221 ms on lock 0x7b54003b1fd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:27.287919 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":508101504}
I20260807 11:10:27.241748 28252 raft_consensus.cc:2781] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:27.292260 25981 leader_election.cc:302] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:27.264484 25872 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:27.294610 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:27.296247 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:27.297060 26033 raft_consensus.cc:2454] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
W20260807 11:10:27.286528 25798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1399) took 1410 ms (client timeout 1562 ms). Trace:
I20260807 11:10:27.266954 28647 raft_consensus.cc:2781] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260807 11:10:27.305178 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 68668.8us
I20260807 11:10:27.305285 28647 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:27.305764 28647 raft_consensus.cc:3037] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:27.306210 25869 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:27.306993 27882 raft_consensus.cc:2781] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:27.307428 27882 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:27.307785 27882 raft_consensus.cc:3037] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:27.307883 28631 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:27.308919 25750 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.120.130:36465 timed out after 1.502s (SENT) [suppressed 1 similar messages]
I20260807 11:10:27.308364 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:27.303485 28354 raft_consensus.cc:2726] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:27.315248 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:27.246332 28676 raft_consensus.cc:2781] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:27.316531 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:27.330822 26038 raft_consensus.cc:2379] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 4: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:27.332448 25869 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:27.333689 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:27.337862 25808 raft_consensus.cc:2379] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 2: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:27.338483 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:27.339200 26038 raft_consensus.cc:2379] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 3: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:27.341286 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:27.297891 28244 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:27.311033 28561 raft_consensus.cc:2781] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:27.242235 28242 consensus_queue.cc:1046] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.078s
I20260807 11:10:27.343897 25752 leader_election.cc:302] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:27.242635 25796 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35"
dest_uuid: "606d49e053a346e486f48cb31832b815"
mode: GRACEFUL
new_leader_uuid: "4fd6381544574ddca559ce63b9243efc"
from {username='slave'} at 127.0.0.1:58114
I20260807 11:10:27.350018 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:27.346164 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:27.348546 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:27.350621 25806 raft_consensus.cc:2454] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:27.351064 26033 raft_consensus.cc:2454] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:27.350488 25796 raft_consensus.cc:595] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Received request to transfer leadership to TS 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:27.317936 28686 raft_consensus.cc:2781] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:27.352049 25796 raft_consensus.cc:603] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 11:10:27.353099 26036 raft_consensus.cc:3037] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 4
I20260807 11:10:27.313297 28631 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:27.353868 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:27.356839 26936 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:27.330714 28676 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:27.359758 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:27.340451 28678 raft_consensus.cc:2726] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:27.363706 25786 raft_consensus.cc:2454] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:27.335373 28143 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:27.359968 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:27.371915 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:27.372305 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:27.345738 28252 raft_consensus.cc:686] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 3 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:27.375128 25800 raft_consensus.cc:2354] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4.
I20260807 11:10:27.375708 25808 raft_consensus.cc:2454] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:10:27.376668 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:27.372383 25981 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:27.349856 28561 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:27.383028 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:27.354281 28457 raft_consensus.cc:2726] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 11:10:27.383755 25752 leader_election.cc:302] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:27.355912 28244 raft_consensus.cc:2781] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260807 11:10:27.372675 28498 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.136s user 0.010s sys 0.006s
W20260807 11:10:27.303494 25798 rpcz_store.cc:269] 0807 11:10:25.875988 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:25.885026 (+ 9038us) service_pool.cc:227] Handling call
0807 11:10:27.285435 (+1400409us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b5800003c98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc46061b0 00007f3fc4605a94 00007f3fbdcedc04 00007f3fbdcedb92
0807 11:10:27.286448 (+ 1013us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":27803904}
I20260807 11:10:27.389196 25869 leader_election.cc:302] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:27.354802 28686 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:27.392196 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:27.394575 25808 raft_consensus.cc:3037] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:27.365921 28488 raft_consensus.cc:482] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:27.401484 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:27.363250 28676 raft_consensus.cc:3037] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:27.410612 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:27.404314 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:27.371762 28634 raft_consensus.cc:2726] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:27.370365 28631 raft_consensus.cc:482] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:27.415707 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:27.376132 28566 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:27.411441 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:27.418741 26034 raft_consensus.cc:2454] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:27.414307 25800 raft_consensus.cc:1226] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260807 11:10:27.387744 28498 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.153s user 0.011s sys 0.006s
I20260807 11:10:27.391178 28561 raft_consensus.cc:3037] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:27.417296 25869 leader_election.cc:398] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Vote denied by peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195) with higher term. Message: Invalid argument: T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for earlier term 4. Current term is 5.
I20260807 11:10:27.421191 25869 leader_election.cc:401] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260807 11:10:27.392649 28386 raft_consensus.cc:2781] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260807 11:10:27.416929 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 108030us
I20260807 11:10:27.389207 28244 raft_consensus.cc:482] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:27.422670 25786 raft_consensus.cc:1261] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:10:27.416155 28354 raft_consensus.cc:2726] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:27.427920 25872 leader_election.cc:302] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:10:27.428260 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 55970.7us
I20260807 11:10:27.403151 28686 raft_consensus.cc:3037] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:27.434811 26036 raft_consensus.cc:2454] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:27.411453 28488 raft_consensus.cc:504] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:27.427772 28244 raft_consensus.cc:3037] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:27.437836 28691 consensus_queue.cc:1046] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260807 11:10:27.438351 28675 raft_consensus.cc:3032] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:27.471863 28675 raft_consensus.cc:729] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:27.472198 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:27.473112 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:27.473645 28691 raft_consensus.cc:3032] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:27.473488 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:27.473955 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:27.474116 28691 raft_consensus.cc:729] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:27.474356 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:27.439188 25981 leader_election.cc:302] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:27.474900 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:27.474964 28691 consensus_queue.cc:260] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:27.439225 28679 log.cc:924] Time spent T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20260807 11:10:27.476364 28691 raft_consensus.cc:3037] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:27.476830 28539 raft_consensus.cc:2726] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:27.477417 28488 leader_election.cc:288] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:27.422166 28386 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:27.478200 28386 raft_consensus.cc:3037] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:27.479694 28205 leader_election.cc:288] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:27.480420 25931 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1474) took 1942 ms (client timeout 1502 ms). Trace:
W20260807 11:10:27.480868 25931 rpcz_store.cc:269] 0807 11:10:25.538302 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:27.480220 (+1941918us) service_pool.cc:214] Skipping call since client already timed out
0807 11:10:27.480342 (+ 122us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260807 11:10:27.481670 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:27.481959 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:27.482299 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:27.482707 25786 raft_consensus.cc:2379] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 5: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
W20260807 11:10:27.440959 28682 log.cc:924] Time spent T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.082s user 0.000s sys 0.001s
I20260807 11:10:27.446416 26035 raft_consensus.cc:1226] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:27.483592 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:27.481983 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:27.485054 25917 raft_consensus.cc:3037] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 3
I20260807 11:10:27.485148 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:27.485852 25920 raft_consensus.cc:2454] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:27.487087 25658 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:40730 (request call id 234) took 6723 ms. Trace:
I20260807 11:10:27.445482 28684 raft_consensus.cc:2726] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 11:10:27.420542 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:27.388993 28106 consensus_queue.cc:1035] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260807 11:10:27.416014 28088 consensus_queue.cc:1046] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260807 11:10:27.448673 28252 consensus_queue.cc:237] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:27.419472 28175 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:27.456230 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:27.457144 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:27.458029 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:27.458066 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:27.487442 25658 rpcz_store.cc:276] 0807 11:10:20.763682 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:20.763861 (+ 179us) service_pool.cc:227] Handling call
0807 11:10:27.486849 (+6722988us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 11:10:21.955209 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 11:10:22.164068 (+208859us) write_op.cc:432] Acquiring schema lock in shared mode
0807 11:10:22.166966 (+ 2898us) write_op.cc:435] Acquired schema lock
0807 11:10:22.185973 (+ 19007us) tablet.cc:660] Decoding operations
0807 11:10:22.465732 (+279759us) write_op.cc:620] Acquiring the partition lock for write op
0807 11:10:22.470793 (+ 5061us) write_op.cc:641] Partition lock acquired for write op
0807 11:10:22.473304 (+ 2511us) tablet.cc:683] Acquiring locks for 7 operations
0807 11:10:22.610018 (+136714us) tablet.cc:699] Row locks acquired
0807 11:10:22.630383 (+ 20365us) write_op.cc:260] PREPARE: finished
0807 11:10:22.659514 (+ 29131us) write_op.cc:270] Start()
0807 11:10:22.674245 (+ 14731us) write_op.cc:276] Timestamp: P: 1786101022652068 usec, L: 0
0807 11:10:22.683822 (+ 9577us) op_driver.cc:331] REPLICATION: starting
0807 11:10:22.741490 (+ 57668us) log.cc:842] Serialized 2925 byte log entry
0807 11:10:24.859298 (+2117808us) op_driver.cc:444] REPLICATION: finished
0807 11:10:25.279397 (+420099us) write_op.cc:301] APPLY: starting
0807 11:10:25.279509 (+ 112us) tablet.cc:1365] starting BulkCheckPresence
0807 11:10:25.279744 (+ 235us) tablet.cc:1368] finished BulkCheckPresence
0807 11:10:25.279887 (+ 143us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 11:10:25.281057 (+ 1170us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 11:10:25.307553 (+ 26496us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0807 11:10:25.316242 (+ 8689us) write_op.cc:312] APPLY: finished
0807 11:10:25.444999 (+128757us) log.cc:842] Serialized 59 byte log entry
0807 11:10:25.493435 (+ 48436us) write_op.cc:489] Releasing partition, row and schema locks
0807 11:10:25.493616 (+ 181us) write_op.cc:454] Released schema lock
0807 11:10:25.493680 (+ 64us) write_op.cc:341] FINISH: Updating metrics
0807 11:10:25.494060 (+ 380us) write_op.cc:489] Releasing partition, row and schema locks
0807 11:10:25.494089 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":381525,"apply.run_cpu_time_us":4391,"apply.run_wall_time_us":215083,"num_ops":7,"prepare.queue_time_us":347236,"prepare.run_cpu_time_us":34218,"prepare.run_wall_time_us":2375230,"raft.queue_time_us":450172,"raft.run_cpu_time_us":67160,"raft.run_wall_time_us":364103,"replication_time_us":2161394,"thread_start_us":2381831,"threads_started":4,"wal-append.queue_time_us":1545042,"wal-append.run_cpu_time_us":40883,"wal-append.run_wall_time_us":2223762}]]}
W20260807 11:10:27.463588 25918 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1461) took 2418 ms (client timeout 2210 ms). Trace:
W20260807 11:10:27.506846 25918 rpcz_store.cc:269] 0807 11:10:25.044596 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:25.138696 (+ 94100us) service_pool.cc:227] Handling call
0807 11:10:27.463325 (+2324629us) spinlock_profiling.cc:243] Waited 958 ms on lock 0x7b54001df454. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:27.463457 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2202670848}
I20260807 11:10:27.464735 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1446) took 1971 ms. Trace:
I20260807 11:10:27.465559 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:27.507771 26046 rpcz_store.cc:276] 0807 11:10:25.493409 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:25.498001 (+ 4592us) service_pool.cc:227] Handling call
0807 11:10:25.498315 (+ 314us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:25.697666 (+199351us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:25.697707 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:25.697748 (+ 41us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:25.697777 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:27.464442 (+1766665us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b540029d954. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:27.464507 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:27.464654 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":355552000}
I20260807 11:10:27.508490 25796 raft_consensus.cc:3037] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
W20260807 11:10:27.442425 28649 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.006s sys 0.012s
W20260807 11:10:27.509351 28649 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.680s user 0.006s sys 0.013s
I20260807 11:10:27.490204 28626 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:27.511520 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1455) took 1778 ms. Trace:
I20260807 11:10:27.511951 26039 rpcz_store.cc:276] 0807 11:10:25.733333 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:25.733459 (+ 126us) service_pool.cc:227] Handling call
0807 11:10:26.076028 (+342569us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:26.108383 (+ 32355us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:27.511347 (+1402964us) spinlock_profiling.cc:243] Waited 323 ms on lock 0x7b5400229f50. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:27.511465 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":743951104}
I20260807 11:10:27.514885 25932 raft_consensus.cc:1226] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260807 11:10:27.467267 28269 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:27.453652 28457 consensus_queue.cc:1046] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:10:27.447686 28503 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:27.440034 27882 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:27.452615 28561 raft_consensus.cc:504] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:27.492223 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 53105.9us
I20260807 11:10:27.540884 25932 raft_consensus.cc:1226] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260807 11:10:27.392510 28385 raft_consensus.cc:504] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:27.544821 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:27.491997 25752 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:10:27.546308 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 75108.8us
I20260807 11:10:27.501231 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:27.425661 28566 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:27.427392 28631 raft_consensus.cc:504] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:27.553577 28503 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:27.554080 28631 leader_election.cc:288] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:27.554867 26985 raft_consensus.cc:1226] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:27.555794 26978 raft_consensus.cc:2354] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5.
I20260807 11:10:27.556273 28566 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:27.502338 28078 raft_consensus.cc:2726] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009) with higher term. Message: Invalid argument: T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 2. Current term is 3.
I20260807 11:10:27.557327 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:27.557585 28618 raft_consensus.cc:2781] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:27.558146 28618 raft_consensus.cc:686] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:27.501977 25655 catalog_manager.cc:5495] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 1 to 2, leader changed from b22d69db45194908ad3d345eba9badd9 (127.23.120.131) to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 2 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:10:27.559072 28618 consensus_queue.cc:237] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:27.503022 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:27.559540 25655 catalog_manager.cc:5495] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 2 to 3, leader changed from 4fd6381544574ddca559ce63b9243efc (127.23.120.130) to eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132). New cstate: current_term: 3 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:27.503811 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:27.468389 28462 raft_consensus.cc:2781] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:27.561865 28462 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:27.562264 28462 raft_consensus.cc:3037] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:27.522562 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:27.537803 28425 consensus_queue.cc:1046] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260807 11:10:27.563887 25798 raft_consensus.cc:2454] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:27.565321 28648 raft_consensus.cc:3032] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:27.565724 28648 raft_consensus.cc:729] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:27.506330 25917 raft_consensus.cc:2454] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:27.472755 28675 consensus_queue.cc:260] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:27.461364 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:27.484423 25916 raft_consensus.cc:2454] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:27.520232 25796 raft_consensus.cc:2454] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:27.520866 25808 raft_consensus.cc:2454] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:27.527287 28647 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:27.511600 28626 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:27.532960 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:27.488607 28567 raft_consensus.cc:3037] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:27.534940 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:27.490198 27960 raft_consensus.cc:3032] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:27.541709 25981 leader_election.cc:302] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:27.547873 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 140806us
I20260807 11:10:27.549090 26046 raft_consensus.cc:1226] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:27.551051 28674 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:27.626717 28674 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:27.629015 28674 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:27.552184 28385 leader_election.cc:288] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:27.558159 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:27.632787 28612 log.cc:924] Time spent T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 1.558s user 0.017s sys 0.017s
I20260807 11:10:27.634250 26975 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 362) took 2217 ms. Trace:
W20260807 11:10:27.551987 26979 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50960 (request call id 304) took 1875 ms (client timeout 1648 ms). Trace:
W20260807 11:10:27.635319 26979 rpcz_store.cc:269] 0807 11:10:25.677039 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:27.551703 (+1874664us) service_pool.cc:214] Skipping call since client already timed out
0807 11:10:27.551857 (+ 154us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260807 11:10:27.634608 26975 rpcz_store.cc:276] 0807 11:10:25.417163 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:25.421202 (+ 4039us) service_pool.cc:227] Handling call
0807 11:10:25.451745 (+ 30543us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:25.452242 (+ 497us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:25.452284 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:25.452897 (+ 613us) log.cc:842] Serialized 26 byte log entry
0807 11:10:25.618219 (+165322us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:25.638623 (+ 20404us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:25.651800 (+ 13177us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:26.981185 (+1329385us) spinlock_profiling.cc:243] Waited 17.5 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:10:27.537665 (+556480us) spinlock_profiling.cc:243] Waited 2.29 ms on lock 0x7b240037d9a8. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc5f4cf4 00007f3fbc5f2dfb 00007f3fbc681559 00007f3fbc6811e5 00007f3fbc68082d 00007f3fbf32f46e 00007f3fbf346c0c 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551
0807 11:10:27.633879 (+ 96214us) raft_consensus.cc:1671] finished
0807 11:10:27.633909 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:27.634054 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6766,"raft.queue_time_us":20647,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":13684,"spinlock_wait_cycles":45424640,"thread_start_us":77474,"threads_started":1,"wal-append.queue_time_us":409037}
I20260807 11:10:27.573504 25803 raft_consensus.cc:2454] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:27.569484 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:27.575208 26033 raft_consensus.cc:1226] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:27.569408 25808 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b"
dest_uuid: "606d49e053a346e486f48cb31832b815"
mode: GRACEFUL
new_leader_uuid: "4fd6381544574ddca559ce63b9243efc"
from {username='slave'} at 127.0.0.1:58114
I20260807 11:10:27.638767 25808 raft_consensus.cc:595] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Received request to transfer leadership to TS 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:27.639123 25808 raft_consensus.cc:603] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 11:10:27.577203 28269 raft_consensus.cc:2726] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195) with higher term. Message: Invalid argument: T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for earlier term 4. Current term is 5.
I20260807 11:10:27.577611 26974 pending_rounds.cc:76] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543: Aborting all ops after (but not including) 4
I20260807 11:10:27.581688 28286 raft_consensus.cc:3032] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:27.667907 28286 raft_consensus.cc:729] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:27.628898 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:27.628665 25655 catalog_manager.cc:5495] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 1 to 2, leader changed from b22d69db45194908ad3d345eba9badd9 (127.23.120.131) to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 2 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } }
I20260807 11:10:27.576130 28675 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:27.643006 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:27.644675 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:27.650367 25801 raft_consensus.cc:1226] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 4. Current term is 5. Ops: []
W20260807 11:10:27.653083 25919 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1425) took 2314 ms (client timeout 2047 ms). Trace:
W20260807 11:10:27.655526 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 65167.3us
I20260807 11:10:27.672873 25796 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 5
I20260807 11:10:27.659083 28385 raft_consensus.cc:2726] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260807 11:10:27.656906 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 115264us
I20260807 11:10:27.658682 25799 raft_consensus.cc:2354] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3.
I20260807 11:10:27.593366 28533 raft_consensus.cc:3037] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:27.661448 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:27.663831 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 80205.5us
I20260807 11:10:27.552030 28676 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:27.667428 26974 pending_rounds.cc:97] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260807 11:10:27.669075 26034 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:27.669868 25655 catalog_manager.cc:5495] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 1 to 2, leader changed from 606d49e053a346e486f48cb31832b815 (127.23.120.129) to eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132). New cstate: current_term: 2 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
W20260807 11:10:27.651674 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 127730us
I20260807 11:10:27.639647 27960 raft_consensus.cc:729] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:27.566743 28648 consensus_queue.cc:260] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:27.672978 25919 rpcz_store.cc:269] 0807 11:10:25.335363 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:25.348301 (+ 12938us) service_pool.cc:227] Handling call
0807 11:10:27.588338 (+2240037us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b5400053054. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:27.649683 (+ 61345us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":584421120}
I20260807 11:10:27.699667 25869 leader_election.cc:302] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:27.689275 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:27.695432 25796 raft_consensus.cc:2454] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
W20260807 11:10:27.837644 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 138087us
I20260807 11:10:27.838759 25869 leader_election.cc:302] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:27.856082 26936 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:27.855232 25655 catalog_manager.cc:5495] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 1 to 2, leader changed from 4fd6381544574ddca559ce63b9243efc (127.23.120.130) to eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132). New cstate: current_term: 2 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:27.804630 25981 leader_election.cc:302] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:27.856642 25869 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:27.859469 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 55096.1us
I20260807 11:10:27.860018 25869 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:27.860329 25981 leader_election.cc:398] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4.
I20260807 11:10:27.864964 26936 leader_election.cc:302] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:27.691279 28462 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:27.868862 28385 raft_consensus.cc:2726] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:27.870551 28462 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:27.687600 27882 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:27.701077 28672 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:27.873099 28672 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:27.874406 25918 raft_consensus.cc:1226] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:27.875254 28672 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:27.875599 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:27.876343 25919 raft_consensus.cc:1226] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:27.697249 28244 raft_consensus.cc:504] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:27.878602 28647 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:27.878852 28244 leader_election.cc:288] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:27.880198 25806 raft_consensus.cc:1226] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 1. Current term is 3. Ops: []
I20260807 11:10:27.882098 25932 raft_consensus.cc:1226] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:27.875629 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:27.889309 25917 raft_consensus.cc:1226] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:27.889973 28538 raft_consensus.cc:1050] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9: attempting to promote NON_VOTER eb1fca62dcb34ed9b32188ccb19f0543 to VOTER
I20260807 11:10:27.890445 25981 leader_election.cc:302] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:27.890654 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:27.892119 28385 raft_consensus.cc:2726] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:27.893386 28423 raft_consensus.cc:2726] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:27.676488 28457 consensus_queue.cc:1225] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer 606d49e053a346e486f48cb31832b815 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260807 11:10:27.839078 28693 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:27.907413 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 47504.1us
I20260807 11:10:27.896045 28561 leader_election.cc:288] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:27.693554 28386 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:27.897828 25926 raft_consensus.cc:1261] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:10:27.899641 25931 raft_consensus.cc:3032] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:27.902674 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:27.895318 28592 raft_consensus.cc:2781] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:10:27.921993 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:27.922582 25918 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:27.692258 28449 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:27.906464 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 45284.3us
I20260807 11:10:27.910816 25869 leader_election.cc:302] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:27.918676 25917 raft_consensus.cc:3032] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:27.926222 25917 raft_consensus.cc:729] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:27.860448 28354 raft_consensus.cc:2726] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 11:10:27.867457 28678 raft_consensus.cc:2726] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 11:10:27.868119 28088 raft_consensus.cc:3037] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:27.701573 28709 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:27.852308 26974 raft_consensus.cc:2866] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:10:27.922987 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:27.944026 25921 raft_consensus.cc:2454] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:27.924243 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:27.944923 26046 raft_consensus.cc:3037] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 3
I20260807 11:10:27.924836 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:27.946069 26038 raft_consensus.cc:3037] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 3
I20260807 11:10:27.921548 25931 raft_consensus.cc:729] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:27.924516 25750 leader_election.cc:398] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5.
I20260807 11:10:27.949330 25750 leader_election.cc:401] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260807 11:10:27.954365 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:27.955037 26041 raft_consensus.cc:1226] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:27.931510 25869 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:27.931596 26034 raft_consensus.cc:2454] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:10:27.932593 25786 raft_consensus.cc:1226] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:27.930991 25917 consensus_queue.cc:260] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:27.788178 28286 consensus_queue.cc:260] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:27.923871 27882 raft_consensus.cc:2726] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:27.938061 28592 raft_consensus.cc:686] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:27.966941 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:27.981451 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:27.988003 25797 raft_consensus.cc:1226] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 2 LEADER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:27.988073 26041 raft_consensus.cc:2454] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:27.989456 25801 raft_consensus.cc:2379] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 3: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:27.936435 28252 raft_consensus.cc:2726] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:27.907171 28498 consensus_queue.cc:1035] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260807 11:10:27.917438 28462 consensus_queue.cc:1046] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20260807 11:10:27.991663 25808 raft_consensus.cc:1226] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:27.992475 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:27.920811 28693 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:27.996076 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:27.997593 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.005584 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:28.006337 25803 raft_consensus.cc:3037] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 4
I20260807 11:10:28.000001 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:28.008914 26035 raft_consensus.cc:1226] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:28.009018 26039 raft_consensus.cc:3037] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:27.980094 28425 raft_consensus.cc:2726] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:27.946768 28354 consensus_queue.cc:1046] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.082s
I20260807 11:10:27.977458 28205 raft_consensus.cc:2726] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5.
I20260807 11:10:28.052584 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:28.066627 25799 raft_consensus.cc:2454] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:27.944443 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:27.892381 28538 consensus_queue.cc:237] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:28.068550 26046 raft_consensus.cc:2454] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:27.986249 28707 consensus_queue.cc:1046] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.072s
I20260807 11:10:28.001996 28488 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:28.071621 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.073050 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:28.073446 26038 raft_consensus.cc:2454] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
W20260807 11:10:28.044509 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 150940us
I20260807 11:10:28.076380 25803 raft_consensus.cc:2454] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:27.947198 27960 consensus_queue.cc:260] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:28.013901 27882 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:28.079174 25919 raft_consensus.cc:3037] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:27.932878 28449 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:28.093158 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.077742 25925 raft_consensus.cc:3037] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 3
I20260807 11:10:28.082020 28676 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:28.095422 28686 consensus_meta.cc:287] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.590s user 0.009s sys 0.000s
I20260807 11:10:28.095535 25932 raft_consensus.cc:2454] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:28.093814 25921 raft_consensus.cc:2454] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:28.095860 28686 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:28.096516 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.097891 27882 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:28.097910 28686 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:28.099454 28488 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:28.096487 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:10:28.092514 28691 consensus_meta.cc:287] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.596s user 0.003s sys 0.003s
I20260807 11:10:28.096972 28700 raft_consensus.cc:482] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:28.122390 28700 raft_consensus.cc:504] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:28.125111 28143 raft_consensus.cc:482] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:28.125754 28143 raft_consensus.cc:504] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:28.109761 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:28.108150 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:28.109774 25919 raft_consensus.cc:2454] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:28.099979 27882 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:28.112396 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.127929 26986 raft_consensus.cc:2454] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:28.130041 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.021708 28252 consensus_queue.cc:1046] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.082s
I20260807 11:10:28.138012 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.138926 25932 raft_consensus.cc:2454] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:28.144721 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.145051 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:28.145387 25932 raft_consensus.cc:2454] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:28.128423 27960 raft_consensus.cc:3037] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:28.157711 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.159355 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.164176 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:28.135958 28700 leader_election.cc:288] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:27.986508 28709 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:28.150564 27882 raft_consensus.cc:3032] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:28.174922 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.180277 25919 raft_consensus.cc:2454] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
W20260807 11:10:28.177968 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 194164us
I20260807 11:10:28.162694 28386 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:28.191540 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.192342 25752 leader_election.cc:302] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:28.161576 26976 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:28.110247 28592 consensus_queue.cc:237] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:28.194388 25921 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:28.196817 25752 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:28.192425 25932 raft_consensus.cc:2454] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:28.204492 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:28.205104 25981 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:28.206010 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:28.206604 25981 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:28.208101 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.208247 25981 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:28.179816 27882 raft_consensus.cc:729] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:28.218657 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.219641 25932 raft_consensus.cc:2454] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:28.220328 25872 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:28.220533 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:28.221218 25932 raft_consensus.cc:2379] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 3: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:28.173125 28675 consensus_queue.cc:1046] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260807 11:10:28.226641 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.227269 25932 raft_consensus.cc:2454] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:28.227917 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:28.230221 26038 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 1.
I20260807 11:10:28.234870 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.196405 28386 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:28.238811 28726 raft_consensus.cc:2781] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:28.238723 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:28.239100 28700 raft_consensus.cc:2781] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:28.239555 25932 raft_consensus.cc:2354] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 2. Current term is 3.
I20260807 11:10:28.239418 28726 raft_consensus.cc:686] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:28.209975 28488 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:28.221904 28385 raft_consensus.cc:2781] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:28.237540 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:28.244290 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:28.222114 25981 leader_election.cc:398] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195) with higher term. Message: Invalid argument: T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4.
I20260807 11:10:28.218693 28686 raft_consensus.cc:2781] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:28.243096 25869 leader_election.cc:302] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:28.244445 25919 raft_consensus.cc:3037] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:28.244941 25981 leader_election.cc:401] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260807 11:10:28.244982 25926 raft_consensus.cc:3037] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:28.224071 28533 raft_consensus.cc:2781] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:28.227408 28693 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:28.258061 25869 leader_election.cc:302] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:28.257474 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.258982 25932 raft_consensus.cc:2362] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5. Re-sending same reply.
I20260807 11:10:28.259686 25869 leader_election.cc:302] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:28.264276 25869 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:28.264961 25918 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:28.265748 25869 leader_election.cc:302] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:28.266026 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.268090 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:28.269483 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:28.270157 25932 raft_consensus.cc:2362] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3. Re-sending same reply.
I20260807 11:10:28.278766 28641 consensus_queue.cc:1046] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 11:10:28.279908 28641 raft_consensus.cc:3032] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:28.280321 28641 raft_consensus.cc:729] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:28.281356 28641 consensus_queue.cc:260] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:28.283064 28641 raft_consensus.cc:3037] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:28.255337 28385 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:28.257174 28533 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:28.256316 28686 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:28.335332 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.260819 28707 raft_consensus.cc:2726] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 11:10:28.260962 28672 raft_consensus.cc:2726] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:28.336884 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.339775 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:28.266840 28626 raft_consensus.cc:2726] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195) with higher term. Message: Invalid argument: T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4.
I20260807 11:10:28.256379 28462 consensus_queue.cc:1046] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260807 11:10:28.344933 26033 raft_consensus.cc:2379] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 4: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:28.294778 28700 raft_consensus.cc:686] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:28.350492 25919 raft_consensus.cc:2454] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:28.294868 28106 raft_consensus.cc:2726] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:28.345038 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:28.360447 26046 raft_consensus.cc:3037] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:28.335597 28533 raft_consensus.cc:3037] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:28.355268 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:28.302099 28385 raft_consensus.cc:3037] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:28.358078 28700 consensus_queue.cc:237] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:28.359757 26039 raft_consensus.cc:2454] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:28.366806 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:10:28.351109 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 130471us
I20260807 11:10:28.335767 28686 raft_consensus.cc:3037] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:28.363916 25932 raft_consensus.cc:3037] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
W20260807 11:10:28.365306 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 99879.3us
I20260807 11:10:28.257192 28386 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:28.370234 25981 leader_election.cc:302] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:28.377875 25919 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455"
dest_uuid: "4fd6381544574ddca559ce63b9243efc"
mode: GRACEFUL
new_leader_uuid: "b22d69db45194908ad3d345eba9badd9"
from {username='slave'} at 127.0.0.1:34992
I20260807 11:10:28.410243 25919 raft_consensus.cc:595] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 3 LEADER]: Received request to transfer leadership to TS b22d69db45194908ad3d345eba9badd9
W20260807 11:10:28.403218 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 137555us
I20260807 11:10:28.406044 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:28.361644 28707 raft_consensus.cc:2726] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:28.412827 25869 leader_election.cc:302] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:28.267760 27882 consensus_queue.cc:260] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:28.417399 25869 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:28.369678 28561 raft_consensus.cc:2741] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
W20260807 11:10:28.421840 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 51686.8us
I20260807 11:10:28.365018 28626 consensus_queue.cc:1035] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20260807 11:10:28.439224 25869 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:28.441143 25869 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:28.441865 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:28.442214 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 81839us
I20260807 11:10:28.442672 25918 raft_consensus.cc:2379] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 2: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:28.422030 28462 raft_consensus.cc:2726] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:28.452033 25932 raft_consensus.cc:2454] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:28.452911 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:28.453516 25932 raft_consensus.cc:3037] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:28.450879 26046 raft_consensus.cc:2454] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:28.435839 28088 raft_consensus.cc:2726] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:28.439667 28707 raft_consensus.cc:2781] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:28.422830 28709 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:28.445210 28106 raft_consensus.cc:2726] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 11:10:28.465590 25920 raft_consensus.cc:1226] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:28.470465 26046 raft_consensus.cc:1261] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 11:10:28.476397 25932 raft_consensus.cc:2454] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
W20260807 11:10:28.477718 28728 log.cc:924] Time spent T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
W20260807 11:10:28.485412 28720 log.cc:924] Time spent T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.141s user 0.000s sys 0.001s
I20260807 11:10:28.485109 28747 consensus_queue.cc:1046] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:28.489566 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.487795 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:28.490592 25918 raft_consensus.cc:2354] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3.
I20260807 11:10:28.494326 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:28.494932 25786 raft_consensus.cc:3037] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:28.496011 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:28.496012 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:28.496433 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:28.497130 25806 raft_consensus.cc:3037] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:28.463536 28707 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:28.489503 25750 leader_election.cc:302] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:28.514053 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.494879 25926 raft_consensus.cc:2454] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:28.440348 28724 consensus_queue.cc:1046] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260807 11:10:28.530298 25797 raft_consensus.cc:1226] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:28.531883 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.532706 25926 raft_consensus.cc:2354] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3.
I20260807 11:10:28.533553 25806 raft_consensus.cc:2454] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
W20260807 11:10:28.534161 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 90132.6us
I20260807 11:10:28.536048 25869 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:28.536307 25796 raft_consensus.cc:1261] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:10:28.539486 25869 leader_election.cc:302] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:28.539608 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:28.541043 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.546763 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:28.547165 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.547843 25918 raft_consensus.cc:2354] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3.
I20260807 11:10:28.549924 26038 raft_consensus.cc:1226] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:28.521149 28707 raft_consensus.cc:3037] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:28.553560 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:28.422096 28726 consensus_queue.cc:237] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:28.521831 28719 raft_consensus.cc:2726] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:28.554874 25872 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:28.557089 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.557298 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.498524 28619 consensus_queue.cc:1046] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20260807 11:10:28.561451 26037 raft_consensus.cc:1261] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:10:28.562975 25918 raft_consensus.cc:2354] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4.
W20260807 11:10:28.561609 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 88958.5us
I20260807 11:10:28.546826 28088 raft_consensus.cc:2726] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:28.464825 28738 consensus_queue.cc:1046] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.086s
I20260807 11:10:28.555639 28740 raft_consensus.cc:2726] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260807 11:10:27.990697 28706 consensus_queue.cc:1159] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b22d69db45194908ad3d345eba9badd9" responder_term: 3 status { last_received { term: 3 index: 2 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 3 index: 2 } }
I20260807 11:10:28.589056 28751 raft_consensus.cc:2726] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:28.557330 28672 consensus_queue.cc:1046] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260807 11:10:28.589854 28674 raft_consensus.cc:3032] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:28.590353 26980 raft_consensus.cc:2454] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:28.592741 28385 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:28.593225 28286 raft_consensus.cc:3037] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:28.593711 28619 raft_consensus.cc:3032] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Stepping down as leader of term 1
W20260807 11:10:28.109702 28670 consensus_queue.cc:1159] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" responder_term: 2 status { last_received { term: 1 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260807 11:10:28.545272 28488 consensus_queue.cc:1046] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260807 11:10:28.559134 28724 consensus_queue.cc:1035] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20260807 11:10:28.560431 28761 consensus_queue.cc:1035] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260807 11:10:28.548506 28759 raft_consensus.cc:2781] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election won for term 3
W20260807 11:10:28.609103 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 119667us
I20260807 11:10:28.613808 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:28.627751 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.628690 25786 raft_consensus.cc:2454] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:28.632412 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:28.633762 25803 raft_consensus.cc:1261] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:10:28.642773 25796 raft_consensus.cc:1226] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260807 11:10:28.646296 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.671140 25921 raft_consensus.cc:2379] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 3: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:28.658910 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.674954 25800 raft_consensus.cc:1226] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:28.667917 25799 raft_consensus.cc:3037] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
W20260807 11:10:28.669152 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 114342us
I20260807 11:10:28.678011 25872 leader_election.cc:302] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:28.664397 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:28.666214 25750 leader_election.cc:302] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:28.680411 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 243221us
I20260807 11:10:28.631759 28252 raft_consensus.cc:2781] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election won for term 4
W20260807 11:10:28.661239 25925 consensus_meta.cc:287] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.567s user 0.010s sys 0.000s
I20260807 11:10:28.686996 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.682981 25981 leader_election.cc:302] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:28.686782 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.688493 26936 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:28.649395 28438 raft_consensus.cc:3032] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:28.663408 28773 raft_consensus.cc:3032] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:28.662386 28385 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:28.643757 28674 raft_consensus.cc:729] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:28.787432 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:28.665680 28759 raft_consensus.cc:686] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:28.656929 28619 raft_consensus.cc:729] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:28.658895 28386 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:28.789759 25981 leader_election.cc:302] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:28.850986 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:10:28.789875 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 111935us
I20260807 11:10:28.646582 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:28.833731 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:10:28.855904 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 189745us
I20260807 11:10:28.687750 28488 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:28.859172 25799 raft_consensus.cc:2454] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:28.790853 28773 raft_consensus.cc:729] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:28.662082 28707 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:28.866462 28488 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:28.867265 26976 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 327) took 1410 ms (client timeout 1542 ms). Trace:
W20260807 11:10:28.867679 26976 rpcz_store.cc:269] 0807 11:10:27.457088 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:27.457583 (+ 495us) service_pool.cc:227] Handling call
0807 11:10:28.867098 (+1409515us) spinlock_profiling.cc:243] Waited 645 ms on lock 0x7b54004c13d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:28.867206 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1482459392}
I20260807 11:10:28.846035 28425 raft_consensus.cc:3032] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:28.784478 28691 raft_consensus.cc:2726] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:28.869423 28707 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:28.869704 28780 raft_consensus.cc:2781] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:28.869908 28425 raft_consensus.cc:729] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:28.871414 28780 raft_consensus.cc:686] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
W20260807 11:10:28.855233 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 65610.1us
I20260807 11:10:28.849387 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:28.789294 28438 raft_consensus.cc:729] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:28.872608 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:28.873126 25981 leader_election.cc:398] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195) with higher term. Message: Invalid argument: T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3.
I20260807 11:10:28.873401 25919 raft_consensus.cc:2379] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 3: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:28.873615 25981 leader_election.cc:401] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260807 11:10:28.873528 28438 consensus_queue.cc:260] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:28.875226 25981 leader_election.cc:302] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:28.857401 25750 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:28.687367 25925 raft_consensus.cc:2454] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:28.644938 28754 consensus_queue.cc:1046] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 2, Time since last communication: 0.217s
I20260807 11:10:28.869938 28778 raft_consensus.cc:2781] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:28.871825 28425 consensus_queue.cc:260] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:28.862850 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:28.868729 28488 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:28.833629 28775 raft_consensus.cc:2726] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:28.839340 28760 raft_consensus.cc:2781] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:28.895918 28773 consensus_queue.cc:260] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:28.919348 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 127739us
W20260807 11:10:28.921890 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 64595.9us
I20260807 11:10:28.922909 25750 leader_election.cc:302] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:28.896021 28759 consensus_queue.cc:237] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:28.898844 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:28.926460 25920 raft_consensus.cc:2379] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 4: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:28.926877 26042 raft_consensus.cc:1226] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:28.927237 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:28.875259 28438 raft_consensus.cc:3037] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:28.934531 25750 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:28.938241 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:28.940618 25920 raft_consensus.cc:2379] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 4: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:28.940945 25919 raft_consensus.cc:2379] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 3: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:28.941807 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:28.888440 28648 raft_consensus.cc:3037] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:28.946441 26033 raft_consensus.cc:1226] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:28.898082 28078 raft_consensus.cc:2781] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:28.951208 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.944043 25750 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:28.891149 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1493) took 1015 ms. Trace:
I20260807 11:10:28.924048 28754 consensus_queue.cc:1225] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer 606d49e053a346e486f48cb31832b815 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260807 11:10:28.919453 28778 raft_consensus.cc:686] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:28.960850 25799 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b8330847089e41b0811e28dc30529234"
dest_uuid: "606d49e053a346e486f48cb31832b815"
mode: GRACEFUL
new_leader_uuid: "b22d69db45194908ad3d345eba9badd9"
from {username='slave'} at 127.0.0.1:58114
I20260807 11:10:28.962636 25799 raft_consensus.cc:595] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Received request to transfer leadership to TS b22d69db45194908ad3d345eba9badd9
I20260807 11:10:28.962985 25799 raft_consensus.cc:603] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 11:10:28.963032 25750 leader_election.cc:302] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:28.899513 28707 consensus_queue.cc:1046] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20260807 11:10:28.939970 28760 raft_consensus.cc:686] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 3 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
W20260807 11:10:28.944728 27960 consensus_meta.cc:287] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.779s user 0.008s sys 0.004s
I20260807 11:10:28.961544 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:28.960206 28670 raft_consensus.cc:2726] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:28.947093 28619 consensus_queue.cc:260] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:28.976065 28778 consensus_queue.cc:237] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:28.976956 28722 raft_consensus.cc:2726] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:28.962574 28780 consensus_queue.cc:237] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:28.945554 28700 raft_consensus.cc:2726] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:28.953929 25925 rpcz_store.cc:276] 0807 11:10:27.875807 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:27.902040 (+ 26233us) service_pool.cc:227] Handling call
0807 11:10:28.889097 (+987057us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:28.968585 25806 raft_consensus.cc:1226] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:28.973251 25786 raft_consensus.cc:1261] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260807 11:10:28.957474 28533 consensus_meta.cc:287] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.576s user 0.009s sys 0.000s
I20260807 11:10:28.951174 28386 raft_consensus.cc:2781] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:28.973201 26936 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:28.950171 28078 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:29.005012 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:28.939145 28665 consensus_queue.cc:1035] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260807 11:10:28.977788 28782 raft_consensus.cc:2726] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195) with higher term. Message: Invalid argument: T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3.
W20260807 11:10:29.014650 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 51690.6us
I20260807 11:10:28.943811 28785 raft_consensus.cc:2781] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:29.024307 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
W20260807 11:10:29.031114 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 91778.9us
I20260807 11:10:29.037427 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:29.060417 25919 raft_consensus.cc:2379] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 5: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
W20260807 11:10:29.048074 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 74962.4us
I20260807 11:10:29.008517 28772 raft_consensus.cc:2781] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:28.947378 28674 consensus_queue.cc:260] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:28.978500 28676 consensus_queue.cc:1046] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20260807 11:10:29.054817 25750 leader_election.cc:302] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:29.062084 26936 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:29.065682 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 190511us
I20260807 11:10:29.066656 26039 raft_consensus.cc:1261] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:10:29.061640 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:10:29.070441 25750 leader_election.cc:302] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:29.051067 28078 raft_consensus.cc:3037] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:29.074985 26936 leader_election.cc:302] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:29.075819 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:29.076613 26046 raft_consensus.cc:2454] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:29.058284 28449 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:29.063556 28252 raft_consensus.cc:686] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:29.091104 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:10:29.064479 28386 raft_consensus.cc:686] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:29.065686 28772 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:29.054279 28787 raft_consensus.cc:2726] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:29.090955 26938 leader_election.cc:302] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:29.091842 25916 raft_consensus.cc:2454] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:29.086212 25750 leader_election.cc:302] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:29.046149 28533 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:29.073547 28079 raft_consensus.cc:2781] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:29.088246 28354 raft_consensus.cc:482] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:29.078336 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:29.108050 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:29.110926 28623 raft_consensus.cc:2781] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:29.111334 28623 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:29.111704 28623 raft_consensus.cc:3037] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:29.112735 28789 raft_consensus.cc:2726] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:29.103271 28079 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:29.115326 28079 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:29.100042 28286 consensus_queue.cc:1046] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20260807 11:10:29.117220 28386 consensus_queue.cc:237] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:29.119001 28797 raft_consensus.cc:3032] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:29.112776 28788 raft_consensus.cc:2726] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 11:10:29.119410 28797 raft_consensus.cc:729] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:29.084887 28782 consensus_queue.cc:1035] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260807 11:10:29.073295 28785 raft_consensus.cc:686] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:29.085506 28760 consensus_queue.cc:237] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:29.122120 28785 consensus_queue.cc:237] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:29.103564 28724 raft_consensus.cc:2781] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:29.125762 28724 raft_consensus.cc:482] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:29.125420 28354 raft_consensus.cc:504] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:29.126245 28724 raft_consensus.cc:3037] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:29.126606 28583 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:29.098027 28772 raft_consensus.cc:3037] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:29.127130 28583 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:29.101045 28676 raft_consensus.cc:3032] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:29.128329 28676 raft_consensus.cc:729] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:29.133111 28676 consensus_queue.cc:260] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:29.134020 28533 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:29.134186 28676 raft_consensus.cc:3037] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:29.135178 26033 raft_consensus.cc:1261] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260807 11:10:29.092803 25981 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:10:29.137610 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 44906.7us
I20260807 11:10:29.138121 25921 raft_consensus.cc:2454] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:29.139068 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:29.139930 25921 raft_consensus.cc:2454] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:10:29.140097 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:29.140724 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:29.141624 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 45688.3us
W20260807 11:10:29.142695 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 57221.8us
I20260807 11:10:29.143520 25750 leader_election.cc:302] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:29.144706 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
W20260807 11:10:29.145200 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 75483.4us
I20260807 11:10:29.145481 25796 raft_consensus.cc:1261] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260807 11:10:29.147188 27960 raft_consensus.cc:2726] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 11:10:29.147228 28533 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:29.151000 28533 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:29.151014 25981 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:29.158133 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:29.161799 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:29.162758 26037 raft_consensus.cc:2379] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 3: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:29.168470 25750 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:29.171593 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 67400.4us
I20260807 11:10:29.173170 28782 raft_consensus.cc:2781] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:29.173759 28782 raft_consensus.cc:686] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:29.174715 28782 consensus_queue.cc:237] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:29.175197 28583 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:29.176393 28583 consensus_queue.cc:1035] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
W20260807 11:10:29.179720 28686 consensus_meta.cc:287] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.783s user 0.009s sys 0.000s
I20260807 11:10:29.176378 28078 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:29.184345 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:29.185683 25799 raft_consensus.cc:3037] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:29.186583 28078 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:29.189685 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 47168.5us
I20260807 11:10:29.191588 25918 raft_consensus.cc:2454] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:29.195236 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:29.195446 25799 raft_consensus.cc:2454] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:29.196558 28798 raft_consensus.cc:2781] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:29.197152 28798 raft_consensus.cc:686] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:29.197745 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:29.198522 25803 raft_consensus.cc:2354] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3.
I20260807 11:10:29.198104 28798 consensus_queue.cc:237] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:29.200539 25981 leader_election.cc:398] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465) with higher term. Message: Invalid argument: T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3.
I20260807 11:10:29.200814 26038 raft_consensus.cc:2354] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 2. Current term is 3.
I20260807 11:10:29.202280 25981 leader_election.cc:302] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:29.180194 28686 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:29.204878 25752 leader_election.cc:398] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Vote denied by peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009) with higher term. Message: Invalid argument: T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 2. Current term is 3.
I20260807 11:10:29.205452 25752 leader_election.cc:401] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260807 11:10:29.205914 26936 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:29.208475 25981 leader_election.cc:302] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:29.209740 28799 raft_consensus.cc:3037] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:29.210176 25981 leader_election.cc:302] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:29.212989 28385 raft_consensus.cc:2726] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:29.216328 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:29.203938 28354 leader_election.cc:288] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:29.220571 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 148944us
I20260807 11:10:29.221702 25869 leader_election.cc:302] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:29.216471 25981 leader_election.cc:398] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465) with higher term. Message: Invalid argument: T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4.
I20260807 11:10:29.224079 25981 leader_election.cc:302] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:29.225328 26038 raft_consensus.cc:1226] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:29.226614 26035 raft_consensus.cc:1226] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 2. Current term is 3. Ops: []
W20260807 11:10:29.228540 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 60229.2us
I20260807 11:10:29.228900 25981 leader_election.cc:302] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:29.216028 28759 raft_consensus.cc:2726] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:29.230979 28649 raft_consensus.cc:2726] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009) with higher term. Message: Invalid argument: T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 2. Current term is 3.
I20260807 11:10:29.231489 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:29.237942 28803 log.cc:924] Time spent T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.078s user 0.000s sys 0.001s
I20260807 11:10:29.238361 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:29.239586 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:29.240459 25925 raft_consensus.cc:2379] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 2: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:29.217898 28787 consensus_queue.cc:1046] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260807 11:10:29.241745 28791 raft_consensus.cc:2741] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260807 11:10:29.234090 25750 leader_election.cc:302] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:29.233923 25869 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:29.215653 28503 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:29.248220 25981 leader_election.cc:398] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 election: Vote denied by peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195) with higher term. Message: Invalid argument: T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3.
I20260807 11:10:29.248601 25981 leader_election.cc:401] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260807 11:10:29.231362 27960 raft_consensus.cc:2726] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:29.235054 28385 raft_consensus.cc:2781] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:29.254668 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:29.261878 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:29.242789 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:29.120478 28797 consensus_queue.cc:260] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:29.231818 28538 raft_consensus.cc:2726] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 11:10:29.264467 25797 raft_consensus.cc:1226] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260807 11:10:29.270330 28623 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:29.215840 28788 consensus_queue.cc:1035] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260807 11:10:29.250587 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:29.251950 25750 leader_election.cc:302] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:29.303764 28761 raft_consensus.cc:2726] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:29.267534 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:29.264279 28800 consensus_queue.cc:1035] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:10:29.267171 28794 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:29.252485 28780 raft_consensus.cc:2726] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195) with higher term. Message: Invalid argument: T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3.
I20260807 11:10:29.267468 28797 raft_consensus.cc:3037] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:29.250977 28759 consensus_queue.cc:1035] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20260807 11:10:29.268936 28804 consensus_queue.cc:1046] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:10:29.261628 28649 raft_consensus.cc:2781] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:29.273674 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:29.273241 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:29.227725 28252 consensus_queue.cc:237] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:29.267200 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:29.265231 25869 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:29.261374 28354 raft_consensus.cc:2726] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:29.269347 28686 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:29.276194 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:29.272398 28623 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:29.325851 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 60725.3us
I20260807 11:10:29.293232 28533 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:29.252600 28385 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:29.333442 25926 raft_consensus.cc:2454] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:29.335863 25869 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:29.446938 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 111147us
I20260807 11:10:29.447517 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:29.447836 25869 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:29.331157 26976 raft_consensus.cc:1261] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEARNER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260807 11:10:29.448814 28533 raft_consensus.cc:2726] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 11:10:29.330472 26974 raft_consensus.cc:2396] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260807 11:10:29.449611 28533 raft_consensus.cc:2726] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 11:10:29.275367 28724 raft_consensus.cc:504] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:29.453290 25808 raft_consensus.cc:1226] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:29.453322 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:29.454974 28724 leader_election.cc:288] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:29.455924 28739 consensus_queue.cc:1035] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260807 11:10:29.436179 28772 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:29.315970 28794 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:29.401224 28385 raft_consensus.cc:3037] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:29.327863 28791 raft_consensus.cc:2726] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:29.330204 25798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1515) took 1363 ms. Trace:
I20260807 11:10:29.310051 28079 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:29.324934 28649 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:29.340879 25925 raft_consensus.cc:2379] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 3: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:29.430835 28686 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:29.459368 25981 leader_election.cc:302] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:29.458791 25869 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:29.438396 28503 raft_consensus.cc:2726] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:29.477164 28649 raft_consensus.cc:3037] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:29.490551 28686 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:29.489396 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:29.493237 25869 leader_election.cc:302] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:29.494720 28623 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:29.495200 28354 raft_consensus.cc:482] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:29.496816 28252 raft_consensus.cc:2726] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:29.497617 25981 leader_election.cc:302] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:29.500411 28785 raft_consensus.cc:3037] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
W20260807 11:10:29.501505 28810 log.cc:924] Time spent T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.234s user 0.001s sys 0.000s
I20260807 11:10:29.502621 28772 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:29.503996 28726 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.855s user 0.010s sys 0.013s
I20260807 11:10:29.509045 28760 raft_consensus.cc:2726] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:29.510972 28623 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:29.511360 28354 raft_consensus.cc:504] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:29.516752 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:29.511035 28759 raft_consensus.cc:3037] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
W20260807 11:10:29.524945 28726 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.876s user 0.010s sys 0.013s
I20260807 11:10:29.525581 25799 raft_consensus.cc:1226] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
W20260807 11:10:29.507706 28808 log.cc:924] Time spent T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.245s user 0.000s sys 0.003s
I20260807 11:10:29.474311 28800 consensus_queue.cc:752] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 2 is ahead of the local log (next sequential op: 2). Destination peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 1.1, Next index: 3, Last known committed idx: 2, Time since last communication: 0.882s
W20260807 11:10:29.529362 26980 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50960 (request call id 313) took 2161 ms (client timeout 1512 ms). Trace:
I20260807 11:10:29.550119 25981 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:29.537923 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:29.534355 28143 leader_election.cc:288] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:29.551985 25925 raft_consensus.cc:3037] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:29.532364 28425 raft_consensus.cc:3037] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:29.552672 26036 raft_consensus.cc:1261] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:10:29.539499 28686 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:10:29.540349 28817 log.cc:924] Time spent T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.078s user 0.000s sys 0.001s
I20260807 11:10:29.542665 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:29.545018 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:29.544953 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:29.532738 28354 leader_election.cc:288] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:29.560746 26041 raft_consensus.cc:2379] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 3: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
W20260807 11:10:29.554177 26980 rpcz_store.cc:269] 0807 11:10:27.366212 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:27.371933 (+ 5721us) service_pool.cc:227] Handling call
0807 11:10:29.514039 (+2142106us) spinlock_profiling.cc:243] Waited 643 ms on lock 0x7b54005dded4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:29.525507 (+ 11468us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1479476992}
I20260807 11:10:29.565227 28794 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:29.567348 28626 raft_consensus.cc:3032] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:29.567881 28626 raft_consensus.cc:729] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:29.569533 28626 consensus_queue.cc:260] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:29.571199 28626 raft_consensus.cc:3037] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:29.571610 25920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1442) took 1153 ms. Trace:
I20260807 11:10:29.572120 25920 rpcz_store.cc:276] 0807 11:10:28.418599 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:28.950912 (+532313us) service_pool.cc:227] Handling call
0807 11:10:29.571398 (+620486us) spinlock_profiling.cc:243] Waited 529 ms on lock 0x7b5400362ed4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:29.571554 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1217137536}
I20260807 11:10:29.574803 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:29.575691 25920 raft_consensus.cc:2354] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3.
I20260807 11:10:29.531431 28623 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:29.572844 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:29.553913 28780 raft_consensus.cc:2726] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:29.557031 28079 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:29.576793 25981 leader_election.cc:398] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465) with higher term. Message: Invalid argument: T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3.
I20260807 11:10:29.576642 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:29.585618 28449 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:29.580338 26037 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:29.592149 25981 leader_election.cc:401] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260807 11:10:29.573252 28759 raft_consensus.cc:2726] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:29.618294 25920 raft_consensus.cc:2362] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3. Re-sending same reply.
I20260807 11:10:29.615170 25872 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:29.651620 28780 raft_consensus.cc:2726] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465) with higher term. Message: Invalid argument: T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3.
I20260807 11:10:29.653245 28739 raft_consensus.cc:482] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
W20260807 11:10:29.654696 28813 log.cc:924] Time spent T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.349s user 0.001s sys 0.000s
I20260807 11:10:29.656378 28778 raft_consensus.cc:482] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:29.678197 28778 raft_consensus.cc:504] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:29.682854 25917 raft_consensus.cc:3037] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
W20260807 11:10:29.687251 28822 log.cc:924] Time spent T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.183s user 0.000s sys 0.002s
I20260807 11:10:29.659519 28449 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:29.690438 28811 log.cc:924] Time spent T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.423s user 0.000s sys 0.002s
I20260807 11:10:29.664685 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:29.691859 26980 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:29.694149 25921 raft_consensus.cc:2454] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:29.694036 28739 raft_consensus.cc:504] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:29.696303 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:29.697674 25921 raft_consensus.cc:2379] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 2: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:29.661669 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:29.698238 25917 raft_consensus.cc:1261] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260807 11:10:29.698863 25920 raft_consensus.cc:2379] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 2: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:29.669230 28760 consensus_queue.cc:752] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 2 is ahead of the local log (next sequential op: 2). Destination peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.401s
I20260807 11:10:29.704188 26046 raft_consensus.cc:1226] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:29.698810 25917 pending_rounds.cc:76] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc: Aborting all ops after (but not including) 1
I20260807 11:10:29.705387 25917 pending_rounds.cc:97] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc: Aborting uncommitted NO_OP operation due to leader change: 2.2
W20260807 11:10:29.665956 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 50912.2us
I20260807 11:10:29.699313 26936 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:29.705753 25917 raft_consensus.cc:2866] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:10:29.706864 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1495) took 1809 ms. Trace:
I20260807 11:10:29.707346 25917 rpcz_store.cc:276] 0807 11:10:27.897478 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:27.903727 (+ 6249us) service_pool.cc:227] Handling call
0807 11:10:27.918563 (+ 14836us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:29.682281 (+1763718us) spinlock_profiling.cc:243] Waited 883 ms on lock 0x7b58000317b0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf63ff63 00007f3fbf6a3aba 00007f3fbf6bb3af 00007f3fbf6bcb4e 00007f3fbf6c6c89 00007f3fbf6c7556 00007f3fbf6c3edf 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684
0807 11:10:29.706534 (+ 24253us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:29.706797 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2030285568}
I20260807 11:10:29.714148 25981 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:29.648136 28794 consensus_queue.cc:1035] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.069s
I20260807 11:10:29.706791 28648 raft_consensus.cc:482] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:29.473245 25798 rpcz_store.cc:276] 0807 11:10:27.966882 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:27.978818 (+ 11936us) service_pool.cc:227] Handling call
0807 11:10:28.072314 (+ 93496us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:28.117842 (+ 45528us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:28.117892 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:28.647793 (+529901us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b2c00010f08. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbe0a7e11 00007f3fbe0b7442 00007f3fbf6ffc11 00007f3fbf6c458a 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772
0807 11:10:28.648359 (+ 566us) log.cc:842] Serialized 26 byte log entry
0807 11:10:29.092814 (+444455us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:29.249671 (+156857us) log.cc:842] Serialized 14 byte log entry
0807 11:10:29.265175 (+ 15504us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:29.305735 (+ 40560us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:29.308987 (+ 3252us) raft_consensus.cc:1671] finished
0807 11:10:29.327321 (+ 18334us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:29.330136 (+ 2815us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11287,"prepare.queue_time_us":216117,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":7243,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":35617,"spinlock_wait_cycles":46717440,"thread_start_us":578235,"threads_started":2,"wal-append.queue_time_us":479408}
I20260807 11:10:29.718611 28648 raft_consensus.cc:504] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:29.720064 28760 raft_consensus.cc:979] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc: : Instructing follower b22d69db45194908ad3d345eba9badd9 to start an election
I20260807 11:10:29.712482 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:29.723134 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:29.723294 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:29.731825 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:29.732630 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:29.736066 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:29.737946 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:29.721371 28252 raft_consensus.cc:2781] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:29.715263 28799 raft_consensus.cc:2726] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:29.721232 28789 raft_consensus.cc:3037] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:29.742316 25981 leader_election.cc:302] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:29.724840 28760 raft_consensus.cc:1067] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 3 LEADER]: Signalling peer b22d69db45194908ad3d345eba9badd9 to start an election
W20260807 11:10:29.727900 28616 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260807 11:10:29.743898 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:29.757177 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:29.758929 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:29.762063 25920 raft_consensus.cc:2379] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 3: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:29.737907 28754 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:29.769083 25920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1443) took 1350 ms. Trace:
I20260807 11:10:29.776389 25920 rpcz_store.cc:276] 0807 11:10:28.418971 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:29.742964 (+1323993us) service_pool.cc:227] Handling call
0807 11:10:29.769018 (+ 26054us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:29.737505 28778 leader_election.cc:288] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:29.746258 28252 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:29.746526 28739 leader_election.cc:288] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:29.841126 28821 log.cc:924] Time spent T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.110s user 0.001s sys 0.000s
W20260807 11:10:29.842891 25916 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1483) took 2170 ms (client timeout 1700 ms). Trace:
W20260807 11:10:29.843307 25916 rpcz_store.cc:269] 0807 11:10:27.672182 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:27.672442 (+ 260us) service_pool.cc:227] Handling call
0807 11:10:29.842710 (+2170268us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54003fedd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:29.842821 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2881297152}
I20260807 11:10:29.844161 28633 raft_consensus.cc:3032] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:29.844516 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:29.844661 28633 raft_consensus.cc:729] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:29.841198 28252 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:29.846143 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:29.845851 28633 consensus_queue.cc:260] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:29.847576 28633 raft_consensus.cc:3037] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:29.772908 28799 raft_consensus.cc:2726] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 11:10:29.853677 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 536780us
I20260807 11:10:29.821924 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:29.857028 25921 raft_consensus.cc:2454] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:29.802543 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:29.758639 28423 raft_consensus.cc:482] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:29.816591 28787 raft_consensus.cc:482] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:29.824113 25931 consensus_queue.cc:260] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:29.829809 28449 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:29.830502 28802 log.cc:924] Time spent T 00000000000000000000000000000000 P 98664bdefe2a4a829847688b8d0d50c1: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260807 11:10:29.800122 28754 raft_consensus.cc:504] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:29.834141 28649 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:29.836513 28385 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:29.775815 25925 raft_consensus.cc:2454] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:10:29.861713 25920 raft_consensus.cc:2454] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:29.891294 25920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1533) took 1269 ms. Trace:
I20260807 11:10:29.891780 25920 rpcz_store.cc:276] 0807 11:10:28.622083 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:29.802194 (+1180111us) service_pool.cc:227] Handling call
0807 11:10:29.891220 (+ 89026us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:29.863859 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:29.865126 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:29.868868 28498 raft_consensus.cc:2726] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:29.872252 25921 raft_consensus.cc:1261] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:10:29.863647 26037 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455"
dest_uuid: "b22d69db45194908ad3d345eba9badd9"
from {username='slave'} at 127.0.0.1:59312
I20260807 11:10:29.875712 28423 raft_consensus.cc:504] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:29.870424 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:29.890545 25931 raft_consensus.cc:3037] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:29.862656 28425 raft_consensus.cc:482] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:29.895084 26033 raft_consensus.cc:3037] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:29.897472 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1517) took 1809 ms. Trace:
I20260807 11:10:29.921468 25921 rpcz_store.cc:276] 0807 11:10:28.087922 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:29.871466 (+1783544us) service_pool.cc:227] Handling call
0807 11:10:29.871842 (+ 376us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:29.896907 (+ 25065us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:29.897397 (+ 490us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:29.923543 25921 raft_consensus.cc:1226] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260807 11:10:29.899556 25917 raft_consensus.cc:2379] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 3: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:29.871531 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:29.880991 28787 raft_consensus.cc:504] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:29.898007 26037 raft_consensus.cc:482] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:10:29.931910 26037 raft_consensus.cc:3037] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:29.909577 28423 leader_election.cc:288] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:29.935379 28837 log.cc:924] Time spent T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260807 11:10:29.937147 25925 raft_consensus.cc:1261] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:10:29.937798 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1438) took 1541 ms. Trace:
I20260807 11:10:29.938145 25925 rpcz_store.cc:276] 0807 11:10:28.396606 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:29.931154 (+1534548us) service_pool.cc:227] Handling call
0807 11:10:29.936685 (+ 5531us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:29.937538 (+ 853us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:29.937739 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:29.910581 28844 raft_consensus.cc:3037] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:29.944006 26038 raft_consensus.cc:1261] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260807 11:10:29.947335 26938 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:29.949481 28643 raft_consensus.cc:2726] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:29.950444 28789 raft_consensus.cc:2726] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:29.924475 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1444) took 1505 ms. Trace:
I20260807 11:10:29.951828 25921 rpcz_store.cc:276] 0807 11:10:28.419277 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:29.922746 (+1503469us) service_pool.cc:227] Handling call
0807 11:10:29.923249 (+ 503us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:29.924133 (+ 884us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:29.924367 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:29.947548 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:29.953342 25921 raft_consensus.cc:1226] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:29.902784 28785 raft_consensus.cc:2726] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:29.956053 25786 raft_consensus.cc:1261] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:10:29.959213 25808 raft_consensus.cc:1261] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:10:29.956739 26936 leader_election.cc:302] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:29.957998 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1451) took 1504 ms. Trace:
I20260807 11:10:29.955798 25919 raft_consensus.cc:2454] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:29.959959 25921 rpcz_store.cc:276] 0807 11:10:28.453800 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:29.952670 (+1498870us) service_pool.cc:227] Handling call
0807 11:10:29.953133 (+ 463us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:29.954338 (+ 1205us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:29.957947 (+ 3609us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:29.961684 25981 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:29.962133 25921 raft_consensus.cc:1261] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:10:29.962915 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1459) took 1369 ms. Trace:
W20260807 11:10:29.963919 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 711886us
I20260807 11:10:29.964156 25921 rpcz_store.cc:276] 0807 11:10:28.593574 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:29.961527 (+1367953us) service_pool.cc:227] Handling call
0807 11:10:29.961783 (+ 256us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:29.962711 (+ 928us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:29.962872 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:29.964893 25750 leader_election.cc:398] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Vote denied by peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465) with higher term. Message: Invalid argument: T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 2. Current term is 3.
I20260807 11:10:29.966168 25800 raft_consensus.cc:1226] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:29.967533 28798 consensus_queue.cc:1046] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:29.967618 25750 leader_election.cc:302] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:29.968393 28798 consensus_queue.cc:1225] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer 4fd6381544574ddca559ce63b9243efc log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260807 11:10:29.968989 28759 raft_consensus.cc:3032] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:29.969311 28759 raft_consensus.cc:729] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:29.970813 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 162) took 1376 ms. Trace:
I20260807 11:10:29.971237 25921 rpcz_store.cc:276] 0807 11:10:28.594200 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:29.965437 (+1371237us) service_pool.cc:227] Handling call
0807 11:10:29.965764 (+ 327us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:29.970487 (+ 4723us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:29.970522 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:29.970563 (+ 41us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:29.970611 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:29.970666 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:29.970762 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4368,"raft.queue_time_us":4472}
W20260807 11:10:29.954515 28782 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.686s user 0.011s sys 0.009s
I20260807 11:10:29.972867 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:29.973676 25786 raft_consensus.cc:2454] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:29.974340 25796 raft_consensus.cc:1261] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:10:29.933279 28794 raft_consensus.cc:482] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:29.980283 25786 raft_consensus.cc:1261] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260807 11:10:29.975410 28782 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.707s user 0.011s sys 0.010s
I20260807 11:10:29.958935 28618 raft_consensus.cc:3032] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:29.971021 28626 raft_consensus.cc:2781] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:29.969915 28643 raft_consensus.cc:2781] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:29.992506 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:29.933072 28425 raft_consensus.cc:504] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:30.006910 28425 leader_election.cc:288] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:29.991149 28794 raft_consensus.cc:504] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:30.008263 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:30.008602 28293 raft_consensus.cc:2781] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:30.009290 28293 raft_consensus.cc:686] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:30.009789 28794 leader_election.cc:288] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:29.995187 28626 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:30.010830 28626 raft_consensus.cc:3037] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:30.011323 28079 consensus_queue.cc:1035] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:10:30.012490 28787 leader_election.cc:288] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:29.991576 28618 raft_consensus.cc:729] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:30.013664 28618 consensus_queue.cc:260] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:30.014727 28618 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:30.016999 28794 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:30.017475 28794 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:30.019408 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:30.020551 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:29.997207 28643 raft_consensus.cc:686] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:29.971830 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1445) took 1552 ms. Trace:
I20260807 11:10:29.972268 28789 consensus_queue.cc:1046] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260807 11:10:30.017612 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:30.022011 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 54453.5us
I20260807 11:10:30.022215 25808 raft_consensus.cc:1261] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260807 11:10:29.998926 28649 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:30.010260 28293 consensus_queue.cc:237] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:30.034171 25750 leader_election.cc:302] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:30.036563 25808 raft_consensus.cc:1226] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:30.037001 28754 leader_election.cc:288] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:29.971613 28739 consensus_queue.cc:1035] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260807 11:10:29.969898 28797 consensus_queue.cc:1035] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260807 11:10:30.041225 28385 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:30.042529 28794 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:30.042783 25931 raft_consensus.cc:1261] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 11:10:30.043522 25931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1499) took 2144 ms. Trace:
I20260807 11:10:30.043987 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:30.043864 25931 rpcz_store.cc:276] 0807 11:10:27.898803 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:27.898964 (+ 161us) service_pool.cc:227] Handling call
0807 11:10:27.899378 (+ 414us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:28.653394 (+754016us) spinlock_profiling.cc:243] Waited 70.2 ms on lock 0x7b540039aba0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf65a951 00007f3fbf6bb321 00007f3fbf6bcb4e 00007f3fbf6c6c89 00007f3fbf6c7556 00007f3fbf6c3edf 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612
0807 11:10:29.823508 (+1170114us) spinlock_profiling.cc:243] Waited 306 ms on lock 0x7b540039aba0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf659dc0 00007f3fbf6bb33a 00007f3fbf6bcb4e 00007f3fbf6c6c89 00007f3fbf6c7556 00007f3fbf6c3edf 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612
0807 11:10:30.043266 (+219758us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:30.043461 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":865202816}
I20260807 11:10:30.044848 28354 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:30.045497 28643 consensus_queue.cc:237] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:30.030015 25925 rpcz_store.cc:276] 0807 11:10:28.419560 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:29.938986 (+1519426us) service_pool.cc:227] Handling call
0807 11:10:29.939345 (+ 359us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:29.939530 (+ 185us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:29.939564 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:29.939608 (+ 44us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:29.941662 (+ 2054us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:29.955280 (+ 13618us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:29.970740 (+ 15460us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:29.993325 28759 consensus_queue.cc:260] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:30.049678 25750 leader_election.cc:302] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:29.995607 28724 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:29.982710 28785 consensus_queue.cc:1035] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20260807 11:10:30.035189 26037 raft_consensus.cc:504] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:30.236439 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:30.220036 25799 raft_consensus.cc:1261] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260807 11:10:30.237586 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 188002us
W20260807 11:10:30.070600 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 91486.8us
I20260807 11:10:30.159533 26033 raft_consensus.cc:2454] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:10:30.095677 25869 leader_election.cc:302] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:30.095959 26936 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:30.239089 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:30.224176 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:30.239500 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 143915us
I20260807 11:10:30.239918 26038 raft_consensus.cc:2454] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:30.241982 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:30.148590 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 170724us
W20260807 11:10:30.246661 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 150807us
I20260807 11:10:30.242578 26038 raft_consensus.cc:2454] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:30.248219 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:30.248204 25981 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:30.252199 26038 raft_consensus.cc:2454] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:10:30.249749 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:30.253211 26938 leader_election.cc:302] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:30.244115 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:30.256251 25981 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:30.253897 26037 leader_election.cc:288] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:30.254493 25872 leader_election.cc:302] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:30.257465 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:30.259761 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:30.060041 28649 raft_consensus.cc:2726] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:30.160131 28449 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:30.241427 28856 raft_consensus.cc:2726] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:30.152675 28789 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:30.090396 28844 raft_consensus.cc:2726] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:30.043998 25798 raft_consensus.cc:1226] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:30.044013 28794 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:30.268194 25750 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:30.247946 28706 raft_consensus.cc:2726] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:30.189369 28806 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:30.196911 28791 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:30.273700 25872 leader_election.cc:302] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:10:30.347718 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 74102.5us
I20260807 11:10:30.348560 25872 leader_election.cc:302] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:30.262925 28561 raft_consensus.cc:2781] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election won for term 3
W20260807 11:10:30.347126 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 82200us
I20260807 11:10:30.332461 28859 raft_consensus.cc:2726] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:30.263325 28425 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:30.251354 28724 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:30.255121 28354 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:30.437952 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 112618us
I20260807 11:10:30.257568 28252 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:30.440272 25916 raft_consensus.cc:2354] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5.
I20260807 11:10:30.441342 28252 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:30.443730 28457 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260807 11:10:30.453199 28734 raft_consensus.cc:2781] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:30.453675 28734 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:30.454041 28734 raft_consensus.cc:3037] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
W20260807 11:10:30.346633 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 78504us
I20260807 11:10:30.260684 28787 raft_consensus.cc:2781] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:30.316696 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:30.327265 28851 raft_consensus.cc:2726] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 11:10:30.353973 25872 leader_election.cc:302] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:30.482177 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 128309us
I20260807 11:10:30.282188 28449 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:30.292387 28782 consensus_queue.cc:1035] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20260807 11:10:30.491317 25750 leader_election.cc:302] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:30.435094 25920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1433) took 2057 ms. Trace:
I20260807 11:10:30.270390 28780 consensus_queue.cc:1046] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.056s
W20260807 11:10:30.432301 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 167404us
I20260807 11:10:30.352283 28286 raft_consensus.cc:2781] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:30.430940 28706 raft_consensus.cc:2781] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:30.319388 28789 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:30.505755 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:30.506628 26034 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:30.434747 25925 raft_consensus.cc:1226] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:30.431371 28561 raft_consensus.cc:686] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:30.348624 28794 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:30.498596 25920 rpcz_store.cc:276] 0807 11:10:28.377318 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:29.899957 (+1522639us) service_pool.cc:227] Handling call
0807 11:10:29.900358 (+ 401us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:29.900794 (+ 436us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:29.900839 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:29.933835 (+ 32996us) log.cc:842] Serialized 26 byte log entry
0807 11:10:30.059206 (+125371us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:30.351422 (+292216us) log.cc:842] Serialized 14 byte log entry
0807 11:10:30.358563 (+ 7141us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:30.358675 (+ 112us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:30.358733 (+ 58us) raft_consensus.cc:1671] finished
0807 11:10:30.358751 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:30.435014 (+ 76263us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":99989,"prepare.queue_time_us":147124,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":2987,"raft.queue_time_us":91250,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":29016,"thread_start_us":241958,"threads_started":2,"wal-append.queue_time_us":89664}
I20260807 11:10:30.458087 28787 raft_consensus.cc:482] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:30.526068 28498 consensus_queue.cc:1035] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:10:30.526260 25750 leader_election.cc:302] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:30.458527 28678 raft_consensus.cc:2726] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:30.529471 25920 raft_consensus.cc:1226] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:30.495373 28867 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:30.497597 28672 raft_consensus.cc:2726] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:30.547255 25920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 166) took 1952 ms. Trace:
W20260807 11:10:30.551164 25926 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1453) took 2064 ms (client timeout 2011 ms). Trace:
I20260807 11:10:30.555315 25920 rpcz_store.cc:276] 0807 11:10:28.595271 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:30.528884 (+1933613us) service_pool.cc:227] Handling call
0807 11:10:30.529301 (+ 417us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:30.546994 (+ 17693us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:30.547196 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:30.557755 25920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 168) took 1962 ms. Trace:
I20260807 11:10:30.558187 25920 rpcz_store.cc:276] 0807 11:10:28.595781 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:30.556711 (+1960930us) service_pool.cc:227] Handling call
0807 11:10:30.557126 (+ 415us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:30.557345 (+ 219us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:30.557383 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:30.557438 (+ 55us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:10:30.557486 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:30.557552 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:30.557690 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:30.353268 28791 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:30.351723 28806 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:30.563791 28806 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:30.555635 25926 rpcz_store.cc:269] 0807 11:10:28.486867 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:28.790575 (+303708us) service_pool.cc:227] Handling call
0807 11:10:30.550916 (+1760341us) spinlock_profiling.cc:243] Waited 424 ms on lock 0x7b5400184a54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:30.551058 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":974927616}
I20260807 11:10:30.505067 28706 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:30.500352 25931 raft_consensus.cc:2454] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:10:30.515231 28649 raft_consensus.cc:2726] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:30.504562 28286 raft_consensus.cc:482] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:30.564431 28806 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:30.503720 27960 raft_consensus.cc:482] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:30.572513 25920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 169) took 1976 ms. Trace:
I20260807 11:10:30.455040 28425 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:30.575835 25926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 170) took 1979 ms. Trace:
I20260807 11:10:30.589636 25926 rpcz_store.cc:276] 0807 11:10:28.596281 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:30.574927 (+1978646us) service_pool.cc:227] Handling call
0807 11:10:30.575245 (+ 318us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:30.575431 (+ 186us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:30.575466 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:30.575504 (+ 38us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:30.575550 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:30.575617 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:30.575766 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:10:30.576821 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 137604us
I20260807 11:10:30.545668 28787 raft_consensus.cc:3037] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:30.549100 28647 raft_consensus.cc:2781] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:30.548702 28780 raft_consensus.cc:3032] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:30.585515 25920 rpcz_store.cc:276] 0807 11:10:28.596025 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:30.559305 (+1963280us) service_pool.cc:227] Handling call
0807 11:10:30.571404 (+ 12099us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:30.571737 (+ 333us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:30.571794 (+ 57us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:30.571879 (+ 85us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:30.571988 (+ 109us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:30.572085 (+ 97us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:30.572411 (+ 326us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:30.576694 28706 raft_consensus.cc:3037] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:30.570076 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 165) took 1975 ms. Trace:
I20260807 11:10:30.583146 28286 raft_consensus.cc:3037] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:30.578101 28672 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:30.588542 26034 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:30.603296 25926 raft_consensus.cc:1226] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:30.607074 25926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 173) took 1987 ms. Trace:
I20260807 11:10:30.607153 26936 leader_election.cc:302] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:30.607709 25926 rpcz_store.cc:276] 0807 11:10:28.619827 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:30.590821 (+1970994us) service_pool.cc:227] Handling call
0807 11:10:30.603035 (+ 12214us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:30.606709 (+ 3674us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:30.606990 (+ 281us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:30.609923 25926 raft_consensus.cc:1261] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:10:30.610780 25926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1480) took 1061 ms. Trace:
I20260807 11:10:30.611239 25926 rpcz_store.cc:276] 0807 11:10:29.549821 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:30.608976 (+1059155us) service_pool.cc:227] Handling call
0807 11:10:30.609457 (+ 481us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:30.610408 (+ 951us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:30.610707 (+ 299us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:30.598738 28780 raft_consensus.cc:729] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:30.570657 28686 consensus_queue.cc:1046] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260807 11:10:30.619767 25920 raft_consensus.cc:1261] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260807 11:10:30.582883 28293 consensus_queue.cc:1035] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260807 11:10:30.620846 25920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1535) took 1998 ms. Trace:
I20260807 11:10:30.596340 28647 raft_consensus.cc:482] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:30.625026 25920 rpcz_store.cc:276] 0807 11:10:28.622654 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:30.603724 (+1981070us) service_pool.cc:227] Handling call
0807 11:10:30.606797 (+ 3073us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:30.620494 (+ 13697us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:30.620741 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:30.628618 25926 raft_consensus.cc:1261] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260807 11:10:30.630842 28858 log.cc:924] Time spent T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.081s user 0.001s sys 0.000s
I20260807 11:10:30.592209 27960 raft_consensus.cc:504] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:30.565629 28867 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:30.631125 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:30.592218 28806 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:30.631943 28863 log.cc:924] Time spent T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
W20260807 11:10:30.645380 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 98723.7us
W20260807 11:10:30.621865 28860 log.cc:924] Time spent T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.230s user 0.001s sys 0.000s
I20260807 11:10:30.649690 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:30.643041 25920 raft_consensus.cc:1261] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260807 11:10:30.650911 25920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1483) took 1043 ms. Trace:
I20260807 11:10:30.632998 25931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 174) took 2012 ms. Trace:
I20260807 11:10:30.630368 28647 raft_consensus.cc:3037] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
W20260807 11:10:30.643877 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 142860us
I20260807 11:10:30.639925 25926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1482) took 1075 ms. Trace:
I20260807 11:10:30.656351 25926 rpcz_store.cc:276] 0807 11:10:29.564562 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:30.627704 (+1063142us) service_pool.cc:227] Handling call
0807 11:10:30.628232 (+ 528us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:30.638645 (+ 10413us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:30.639844 (+ 1199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":398208}
I20260807 11:10:30.651517 25920 rpcz_store.cc:276] 0807 11:10:29.607534 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:30.639581 (+1032047us) service_pool.cc:227] Handling call
0807 11:10:30.642590 (+ 3009us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:30.650489 (+ 7899us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:30.650826 (+ 337us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:30.604575 25925 rpcz_store.cc:276] 0807 11:10:28.595012 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:30.266938 (+1671926us) service_pool.cc:227] Handling call
0807 11:10:30.350311 (+ 83373us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:30.519037 (+168726us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:30.565603 (+ 46566us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:30.616230 28672 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:30.640096 28874 consensus_queue.cc:1035] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260807 11:10:30.661448 28498 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:30.643893 28879 raft_consensus.cc:2726] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:30.668887 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:30.669709 26976 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:30.661988 28498 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:30.671003 28884 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:30.671463 28648 leader_election.cc:288] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:30.671550 28498 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:30.666662 25981 leader_election.cc:302] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:30.616876 28561 consensus_queue.cc:237] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:30.649286 28724 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:30.661718 28425 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:30.688031 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 80967.3us
W20260807 11:10:30.689117 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 338230us
I20260807 11:10:30.692714 25872 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:30.637573 28780 consensus_queue.cc:260] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:30.693461 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 163418us
I20260807 11:10:30.693682 25808 raft_consensus.cc:1226] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:30.704871 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:30.706065 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:30.682119 28789 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:30.707897 25750 leader_election.cc:302] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:30.671491 28884 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:30.711244 28724 raft_consensus.cc:482] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:30.709152 28488 consensus_queue.cc:1046] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20260807 11:10:30.713722 28881 consensus_queue.cc:1046] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260807 11:10:30.652930 28882 consensus_queue.cc:1035] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.067s
I20260807 11:10:30.720135 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 256) took 1152 ms. Trace:
I20260807 11:10:30.720890 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:30.721179 27960 leader_election.cc:288] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:30.704259 28449 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:30.689803 28794 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:30.653178 25931 rpcz_store.cc:276] 0807 11:10:28.620167 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:30.596273 (+1976106us) service_pool.cc:227] Handling call
0807 11:10:30.632271 (+ 35998us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:30.632498 (+ 227us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:30.632534 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:30.632592 (+ 58us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:30.632679 (+ 87us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:30.632781 (+ 102us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:30.632933 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:30.723479 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:30.723834 28538 consensus_queue.cc:1035] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 11:10:30.718073 28724 raft_consensus.cc:504] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:30.725190 28806 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:30.720525 26046 rpcz_store.cc:276] 0807 11:10:29.567811 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:29.568008 (+ 197us) service_pool.cc:227] Handling call
0807 11:10:29.568364 (+ 356us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:29.704943 (+136579us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:30.719911 (+1014968us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b540029ed54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:30.720078 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":297038208}
I20260807 11:10:30.336567 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:10:30.721557 26039 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
W20260807 11:10:30.734520 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 54195.5us
I20260807 11:10:30.734691 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:30.746783 28885 raft_consensus.cc:2781] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:30.747498 28885 raft_consensus.cc:482] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:30.747259 28879 consensus_queue.cc:1046] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 11:10:30.750023 26037 raft_consensus.cc:1226] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:30.748013 28885 raft_consensus.cc:3037] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:30.757335 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:30.760380 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:30.770519 26037 raft_consensus.cc:1261] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:10:30.749576 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:10:30.773984 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 45787.7us
I20260807 11:10:30.726239 28488 raft_consensus.cc:3032] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:30.767890 28286 raft_consensus.cc:504] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:30.682986 28871 consensus_queue.cc:1035] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20260807 11:10:30.778730 26986 raft_consensus.cc:1261] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260807 11:10:30.783618 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:30.726328 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:30.779884 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:30.787539 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:30.788400 25808 raft_consensus.cc:2396] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260807 11:10:30.784287 25797 raft_consensus.cc:2454] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
W20260807 11:10:30.770612 28626 consensus_meta.cc:287] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.759s user 0.013s sys 0.000s
I20260807 11:10:30.755052 28882 raft_consensus.cc:2781] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:10:30.751596 28244 raft_consensus.cc:2726] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 11:10:30.785406 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:30.733510 28791 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:30.790202 25869 leader_election.cc:302] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:30.798779 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:30.799479 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:30.828864 26986 raft_consensus.cc:2396] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260807 11:10:30.778167 28892 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:30.847555 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:30.852326 28892 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:30.855067 28892 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:30.849121 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 59020.5us
I20260807 11:10:30.858469 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:30.858954 25869 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:30.778491 28488 raft_consensus.cc:729] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:30.861222 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:30.868993 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 90761.6us
I20260807 11:10:30.883965 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:30.887753 25869 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:30.891969 28561 raft_consensus.cc:482] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:30.892004 28884 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:30.892943 28561 raft_consensus.cc:504] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:30.809643 26980 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:30.876889 28425 raft_consensus.cc:2781] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:30.877552 28724 leader_election.cc:288] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:30.920701 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 211140us
I20260807 11:10:30.807020 28626 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:30.769567 28891 consensus_queue.cc:1035] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.202s
I20260807 11:10:30.926162 28706 raft_consensus.cc:504] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:30.915185 28425 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:30.914312 28791 raft_consensus.cc:2726] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:30.912818 28897 raft_consensus.cc:2726] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:30.920791 28867 leader_election.cc:288] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:10:30.970232 28903 log.cc:924] Time spent T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.052s user 0.002s sys 0.000s
I20260807 11:10:30.966604 28488 consensus_queue.cc:260] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:30.964938 26979 raft_consensus.cc:1226] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 1. Current term is 2. Ops: []
W20260807 11:10:31.018289 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 158459us
I20260807 11:10:31.039270 26034 raft_consensus.cc:1226] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:30.971735 28425 raft_consensus.cc:3037] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:30.972039 28882 raft_consensus.cc:686] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 5 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:31.060691 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:31.066511 26046 raft_consensus.cc:2454] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:30.930182 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:31.068187 25752 leader_election.cc:302] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:31.081734 26985 raft_consensus.cc:1261] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 11:10:31.082156 26974 raft_consensus.cc:2932] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:31.057613 28293 consensus_queue.cc:1035] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
W20260807 11:10:31.088737 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 177706us
I20260807 11:10:31.069317 28244 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:31.091786 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:31.067710 28561 leader_election.cc:288] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:31.096697 28626 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:31.069157 28789 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:31.075917 28488 raft_consensus.cc:3037] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:31.073426 28911 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:31.098508 25786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1537) took 1083 ms. Trace:
I20260807 11:10:31.087718 25798 raft_consensus.cc:3037] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:31.081018 28286 leader_election.cc:288] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:31.104416 25786 rpcz_store.cc:276] 0807 11:10:30.014694 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:30.015467 (+ 773us) service_pool.cc:227] Handling call
0807 11:10:31.091624 (+1076157us) spinlock_profiling.cc:243] Waited 2.57 ms on lock 0x7b5800003c98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc46061b0 00007f3fc4605a94 00007f3fbdcedc04 00007f3fbdcedb92
0807 11:10:31.098424 (+ 6800us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5900672}
I20260807 11:10:31.103282 28789 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:31.115286 28789 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:31.100142 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:31.098013 28891 raft_consensus.cc:2781] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:31.096714 28773 raft_consensus.cc:3037] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:31.033578 28899 consensus_queue.cc:1035] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.317s
I20260807 11:10:31.110486 28686 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:31.135578 28891 raft_consensus.cc:482] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:31.099709 28244 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:31.185356 28706 leader_election.cc:288] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:31.186300 28286 raft_consensus.cc:482] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:31.187935 28867 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:31.180382 28882 consensus_queue.cc:237] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:31.195608 28891 raft_consensus.cc:3037] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
W20260807 11:10:31.218194 26976 consensus_meta.cc:287] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.548s user 0.003s sys 0.010s
I20260807 11:10:31.218698 26976 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
W20260807 11:10:31.252302 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 364594us
I20260807 11:10:31.116717 28911 raft_consensus.cc:504] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:31.269341 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:31.270354 26038 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 1.
I20260807 11:10:31.199296 28686 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:31.250026 28538 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:31.260454 28911 leader_election.cc:288] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:31.311745 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:31.311806 25750 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.010s sys 0.058s
W20260807 11:10:31.313745 25750 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.533s user 0.011s sys 0.058s
I20260807 11:10:31.214049 28286 raft_consensus.cc:504] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:31.320781 26038 raft_consensus.cc:2379] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 3: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:31.322101 25869 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:31.215082 28867 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:31.310289 26978 raft_consensus.cc:1226] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:31.358856 25750 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:31.360554 28746 consensus_queue.cc:1035] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 11:10:31.368461 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:31.335426 28897 raft_consensus.cc:2781] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:31.313089 28538 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:31.399493 28286 leader_election.cc:288] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:31.385493 28897 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:31.400593 26035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1491) took 3119 ms. Trace:
I20260807 11:10:31.407222 26035 rpcz_store.cc:276] 0807 11:10:28.273996 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:28.274451 (+ 455us) service_pool.cc:227] Handling call
0807 11:10:28.338383 (+ 63932us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:29.227036 (+888653us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:31.392859 (+2165823us) spinlock_profiling.cc:243] Waited 888 ms on lock 0x7b540029d954. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:31.393034 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2042329856}
I20260807 11:10:31.373718 28648 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:31.413652 28914 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:31.415019 28244 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:31.424700 25801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 217) took 2147 ms. Trace:
I20260807 11:10:31.401304 28747 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:31.401398 28746 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:31.425200 25801 rpcz_store.cc:276] 0807 11:10:29.277649 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:29.325169 (+ 47520us) service_pool.cc:227] Handling call
0807 11:10:29.325469 (+ 300us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:29.855472 (+530003us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:29.855499 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:29.869537 (+ 14038us) log.cc:842] Serialized 26 byte log entry
0807 11:10:30.878163 (+1008626us) spinlock_profiling.cc:243] Waited 6.45 ms on lock 0x7b18003beea0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbc77e345 00007f3fc55b5a39 00007f3fbc79b8bb 00007f3fbc799cbf 00007f3fbc79be22 00007f3fbf3455bb 00007f3fbf34b063 00007f3fbf34b411 00007f3fbf6985cf
0807 11:10:31.028140 (+149977us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:10:31.028193 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:31.028295 (+ 102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:31.416408 (+388113us) raft_consensus.cc:1671] finished
0807 11:10:31.416435 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:31.424617 (+ 8182us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":551547,"raft.queue_time_us":542405,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":7455,"spinlock_wait_cycles":14844672,"thread_start_us":149706,"threads_started":1,"wal-append.queue_time_us":1446991}
I20260807 11:10:31.433779 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:31.434554 25801 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
W20260807 11:10:31.437121 28900 log.cc:924] Time spent T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.247s user 0.001s sys 0.000s
I20260807 11:10:31.439050 26036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1523) took 1195 ms. Trace:
I20260807 11:10:31.440618 25869 leader_election.cc:302] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:31.441576 26936 leader_election.cc:302] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:31.444461 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:31.409364 28897 raft_consensus.cc:3037] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:31.416241 28911 raft_consensus.cc:2726] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:31.419646 28078 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:31.470675 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:31.439452 26036 rpcz_store.cc:276] 0807 11:10:30.243237 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:30.243370 (+ 133us) service_pool.cc:227] Handling call
0807 11:10:30.266697 (+ 23327us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:30.484543 (+217846us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:30.484587 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:30.502227 (+ 17640us) log.cc:842] Serialized 26 byte log entry
0807 11:10:30.881793 (+379566us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:10:30.881829 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:30.881896 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:31.438771 (+556875us) raft_consensus.cc:1671] finished
0807 11:10:31.438807 (+ 36us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:31.438982 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7336,"raft.queue_time_us":24854,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":4735,"thread_start_us":588831,"threads_started":2,"wal-append.queue_time_us":595241}
I20260807 11:10:31.414415 28914 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:31.444694 28647 consensus_meta.cc:287] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.779s user 0.012s sys 0.001s
I20260807 11:10:31.485146 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:31.449333 28286 raft_consensus.cc:2726] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260807 11:10:31.489007 25918 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1456) took 2933 ms (client timeout 1948 ms). Trace:
W20260807 11:10:31.492420 25918 rpcz_store.cc:269] 0807 11:10:28.555276 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:28.849115 (+293839us) service_pool.cc:227] Handling call
0807 11:10:31.488707 (+2639592us) spinlock_profiling.cc:243] Waited 236 ms on lock 0x7b54003b24d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:31.488904 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":543084288}
I20260807 11:10:31.451536 28789 raft_consensus.cc:2781] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:31.509696 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:31.416056 28648 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:31.511497 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:31.514393 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:31.516534 25918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:39730 (request call id 687) took 3239 ms. Trace:
I20260807 11:10:31.517002 25918 rpcz_store.cc:276] 0807 11:10:28.277241 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:31.515207 (+3237966us) service_pool.cc:227] Handling call
0807 11:10:31.516397 (+ 1190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:31.518663 28789 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:31.519306 28789 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:31.526973 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:31.509912 28885 consensus_meta.cc:287] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.734s user 0.013s sys 0.002s
I20260807 11:10:31.445638 28746 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:31.542052 28930 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:31.546648 28633 raft_consensus.cc:3032] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Stepping down as leader of term 2
W20260807 11:10:31.560628 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 764208us
I20260807 11:10:31.487514 28647 raft_consensus.cc:504] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:31.578918 25872 leader_election.cc:302] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:31.447001 28747 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:31.568193 28686 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:31.602864 28920 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:31.598532 28286 raft_consensus.cc:482] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:31.605420 28286 raft_consensus.cc:504] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:31.585624 28914 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:31.586664 28648 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:31.587384 28740 consensus_queue.cc:1046] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:10:31.480482 28078 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:31.587917 28693 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:31.589640 28425 consensus_meta.cc:287] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.520s user 0.010s sys 0.002s
I20260807 11:10:31.591686 28676 raft_consensus.cc:482] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:31.591828 28438 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:31.577749 28633 raft_consensus.cc:729] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:31.601488 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:31.589622 27882 raft_consensus.cc:3037] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:31.623377 25801 raft_consensus.cc:2454] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:31.626025 28616 raft_consensus.cc:2726] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:31.680755 28438 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:31.693763 28438 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:31.612713 28286 leader_election.cc:288] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:31.599581 28867 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195)
I20260807 11:10:31.688751 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:31.603659 28920 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:31.621567 28791 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:31.575292 28885 raft_consensus.cc:504] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:31.698666 28438 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:31.700506 25801 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
W20260807 11:10:31.705924 25917 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1476) took 2475 ms (client timeout 2248 ms). Trace:
I20260807 11:10:31.721032 28791 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:31.724315 25917 rpcz_store.cc:269] 0807 11:10:29.230693 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:29.739279 (+508586us) service_pool.cc:227] Handling call
0807 11:10:31.705705 (+1966426us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b5400407254. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:31.705835 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":316487552}
I20260807 11:10:31.623175 25798 raft_consensus.cc:2454] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:31.740136 25796 raft_consensus.cc:1226] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260807 11:10:31.634977 28648 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:31.742765 25801 raft_consensus.cc:1226] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:31.729142 28438 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:31.618707 28930 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:31.697032 28538 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:31.752789 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:31.747777 25918 raft_consensus.cc:2354] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4.
I20260807 11:10:31.749979 28438 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:31.751852 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 164) took 3157 ms. Trace:
I20260807 11:10:31.730127 28867 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:31.743840 28787 consensus_meta.cc:287] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 1.140s user 0.014s sys 0.002s
I20260807 11:10:31.755349 25921 rpcz_store.cc:276] 0807 11:10:28.594712 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:29.973455 (+1378743us) service_pool.cc:227] Handling call
0807 11:10:29.974118 (+ 663us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:30.033504 (+ 59386us) raft_consensus.cc:1520] Early marking committed up to index 5
0807 11:10:30.060534 (+ 27030us) log.cc:842] Serialized 14 byte log entry
0807 11:10:30.456983 (+396449us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:30.457032 (+ 49us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:30.457076 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:31.751622 (+1294546us) spinlock_profiling.cc:243] Waited 58.2 ms on lock 0x7b540033a9d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:31.751665 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:31.751798 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":53920,"prepare.queue_time_us":34472,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":36466,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":3501,"spinlock_wait_cycles":133938560,"thread_start_us":363227,"threads_started":2,"wal-append.queue_time_us":207778,"wal-append.run_cpu_time_us":1314,"wal-append.run_wall_time_us":1003333}
I20260807 11:10:31.638276 28693 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:31.776638 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:31.642467 28676 raft_consensus.cc:504] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:31.771340 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:31.640694 28425 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:31.787010 28777 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:31.787128 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:31.787842 26986 raft_consensus.cc:2454] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:31.776304 28804 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:31.800537 25921 raft_consensus.cc:2396] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 11:10:31.826470 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:31.800446 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:31.795598 28920 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:31.794960 28804 raft_consensus.cc:504] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:31.748785 28648 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:31.831564 28746 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:31.832338 28746 raft_consensus.cc:482] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:31.833004 28746 raft_consensus.cc:504] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:31.833891 28648 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:31.835400 25916 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1477) took 2589 ms (client timeout 3281 ms). Trace:
I20260807 11:10:31.795318 28747 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:31.835840 25916 rpcz_store.cc:269] 0807 11:10:29.245536 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:29.845812 (+600276us) service_pool.cc:227] Handling call
0807 11:10:31.835173 (+1989361us) spinlock_profiling.cc:243] Waited 593 ms on lock 0x7b54003b1fd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:31.835336 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1364379392}
W20260807 11:10:31.800010 26974 consensus_meta.cc:287] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.714s user 0.009s sys 0.005s
I20260807 11:10:31.808009 28647 leader_election.cc:288] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:31.812573 28885 leader_election.cc:288] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:31.810303 28735 consensus_queue.cc:1035] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.023s
W20260807 11:10:31.806319 28734 consensus_meta.cc:287] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 1.352s user 0.009s sys 0.009s
I20260807 11:10:31.851653 28804 leader_election.cc:288] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:31.780280 28787 raft_consensus.cc:504] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:31.808166 28791 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:31.818782 28934 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:31.821856 28672 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:31.766532 28633 consensus_queue.cc:260] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:31.790364 28618 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
W20260807 11:10:31.817324 28891 consensus_meta.cc:287] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.600s user 0.008s sys 0.007s
I20260807 11:10:31.787859 28777 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:31.776541 28867 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:31.853826 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:31.851284 28734 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:31.857712 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:31.868526 25917 raft_consensus.cc:2454] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:31.858502 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:31.841445 28946 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:31.842774 28746 leader_election.cc:288] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:31.872376 28633 raft_consensus.cc:3037] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:31.843466 28930 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:31.875053 28676 leader_election.cc:288] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:31.860430 28078 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:31.868377 28773 consensus_meta.cc:287] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.692s user 0.004s sys 0.011s
I20260807 11:10:31.879824 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:31.880219 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:31.881130 25917 raft_consensus.cc:2354] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 2. Current term is 3.
I20260807 11:10:31.882758 28425 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:31.885987 25803 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:31.884040 28946 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:31.886567 28856 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:31.894546 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:31.895793 25916 raft_consensus.cc:2454] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:31.897305 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:31.897565 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:31.898214 25917 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:10:31.901518 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:31.903429 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:31.904083 25917 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:31.873952 25921 raft_consensus.cc:2454] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:31.906001 25800 raft_consensus.cc:1226] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:31.906772 25800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1538) took 1891 ms. Trace:
I20260807 11:10:31.906634 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:31.907179 25800 rpcz_store.cc:276] 0807 11:10:30.014947 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:30.016908 (+ 1961us) service_pool.cc:227] Handling call
0807 11:10:31.905558 (+1888650us) spinlock_profiling.cc:243] Waited 3.28 ms on lock 0x7b5800003c98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc461a2b0 00007f3fc46057f9 00007f3fc46052a2 00007f3fbdceb684
0807 11:10:31.905801 (+ 243us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:31.906496 (+ 695us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:31.906713 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7543040}
I20260807 11:10:31.907794 25921 raft_consensus.cc:3037] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:31.877678 28777 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:31.870728 28891 raft_consensus.cc:504] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:31.916710 25919 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1460) took 3322 ms (client timeout 2161 ms). Trace:
I20260807 11:10:31.917584 28894 raft_consensus.cc:3032] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:31.888250 28856 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:31.873212 28618 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:31.942420 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:31.924672 25919 rpcz_store.cc:269] 0807 11:10:28.593911 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:29.061002 (+467091us) service_pool.cc:227] Handling call
0807 11:10:29.516565 (+455563us) spinlock_profiling.cc:243] Waited 3.31 ms on lock 0x7b58002e0e98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc46061b0 00007f3fc4605a94 00007f3fbdcedc04 00007f3fbdcedb92
0807 11:10:31.916506 (+2399941us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x7b54003b29d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:31.916630 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":737072384}
I20260807 11:10:31.944519 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:31.945281 25919 raft_consensus.cc:3032] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:31.945652 25919 raft_consensus.cc:729] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
W20260807 11:10:31.931197 28935 log.cc:924] Time spent T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.557s user 0.005s sys 0.000s
I20260807 11:10:31.876979 28747 raft_consensus.cc:2932] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:31.868635 28934 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:31.932148 28782 raft_consensus.cc:3032] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:31.934047 28867 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:31.935853 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:31.958992 26036 raft_consensus.cc:2454] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:31.960842 25917 raft_consensus.cc:2454] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
W20260807 11:10:31.948716 26037 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.632s user 0.043s sys 0.064s
I20260807 11:10:31.922878 28354 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:31.919576 28693 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:31.961171 26938 leader_election.cc:302] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:10:31.966802 26037 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.656s user 0.043s sys 0.064s
I20260807 11:10:31.946553 25919 consensus_queue.cc:260] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:31.969440 25919 raft_consensus.cc:3037] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:31.949194 28894 raft_consensus.cc:729] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:31.972726 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:31.974406 26937 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.866s user 0.007s sys 0.038s
I20260807 11:10:31.975263 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:31.975919 26937 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.906s user 0.008s sys 0.039s
I20260807 11:10:31.976256 25916 raft_consensus.cc:2396] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 for term 6 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260807 11:10:31.976758 28891 leader_election.cc:288] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:31.977295 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:31.957078 28782 raft_consensus.cc:729] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:31.980144 28867 raft_consensus.cc:2781] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:31.978302 28894 consensus_queue.cc:260] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:31.989534 28787 leader_election.cc:288] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:31.989214 28867 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:31.994000 28867 raft_consensus.cc:3037] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
W20260807 11:10:31.993323 28897 consensus_meta.cc:287] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.517s user 0.000s sys 0.015s
I20260807 11:10:31.991894 25799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1547) took 1343 ms. Trace:
I20260807 11:10:31.999001 25799 rpcz_store.cc:276] 0807 11:10:30.648448 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:30.648849 (+ 401us) service_pool.cc:227] Handling call
0807 11:10:30.649193 (+ 344us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:30.651622 (+ 2429us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:30.651662 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:30.651692 (+ 30us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:10:30.651713 (+ 21us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:31.991574 (+1339861us) spinlock_profiling.cc:243] Waited 2.19 ms on lock 0x7b54000e42d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:31.991636 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:31.991826 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5041536}
W20260807 11:10:32.001199 28925 log.cc:924] Time spent T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.566s user 0.000s sys 0.002s
I20260807 11:10:32.015825 25808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1552) took 1233 ms. Trace:
I20260807 11:10:32.011521 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:32.014014 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:32.025115 26037 raft_consensus.cc:3037] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:31.990468 28709 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:32.019402 25808 rpcz_store.cc:276] 0807 11:10:30.782631 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:30.864040 (+ 81409us) service_pool.cc:227] Handling call
0807 11:10:30.864321 (+ 281us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:30.885093 (+ 20772us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:30.885130 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:30.897255 (+ 12125us) log.cc:842] Serialized 26 byte log entry
0807 11:10:31.216419 (+319164us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:10:31.216476 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:31.216571 (+ 95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:31.934558 (+717987us) spinlock_profiling.cc:243] Waited 4.62 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:10:32.009635 (+ 75077us) raft_consensus.cc:1671] finished
0807 11:10:32.009665 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:32.015750 (+ 6085us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13929,"raft.queue_time_us":103285,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":18729,"spinlock_wait_cycles":10622720,"thread_start_us":251759,"threads_started":1,"wal-append.queue_time_us":452354}
W20260807 11:10:32.035099 26033 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43246 (request call id 251) took 2762 ms (client timeout 1614 ms). Trace:
I20260807 11:10:32.036960 26042 raft_consensus.cc:1261] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260807 11:10:32.011322 28488 consensus_meta.cc:287] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.906s user 0.016s sys 0.000s
W20260807 11:10:32.040424 26033 rpcz_store.cc:269] 0807 11:10:29.272961 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:29.273109 (+ 148us) service_pool.cc:227] Handling call
0807 11:10:32.034835 (+2761726us) spinlock_profiling.cc:243] Waited 574 ms on lock 0x7b54000bbb54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:32.034982 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1320795776}
I20260807 11:10:32.045913 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:32.012347 28788 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:32.040424 25919 raft_consensus.cc:2454] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:32.043063 28709 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:32.023175 28778 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:32.062698 28789 consensus_meta.cc:287] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.543s user 0.013s sys 0.000s
I20260807 11:10:32.063639 28782 consensus_queue.cc:260] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:32.065785 28618 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:32.017218 28734 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:32.021612 28934 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:32.127269 28619 raft_consensus.cc:3037] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:32.083148 25869 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:32.083297 28616 raft_consensus.cc:482] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:32.083969 28778 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:32.085282 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:32.153703 25796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1535) took 2139 ms. Trace:
I20260807 11:10:32.154615 28778 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:32.089202 25921 raft_consensus.cc:2454] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:32.010337 28897 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:32.088250 28788 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:32.156956 26033 raft_consensus.cc:1226] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:32.157991 25917 raft_consensus.cc:1226] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260807 11:10:32.101894 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:32.112093 28789 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:32.128957 28782 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:32.170433 28897 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:32.131518 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 48459.8us
W20260807 11:10:32.133009 28942 log.cc:924] Time spent T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.761s user 0.000s sys 0.002s
I20260807 11:10:32.176911 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:32.134629 25932 raft_consensus.cc:2454] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:32.185355 26986 raft_consensus.cc:3037] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
W20260807 11:10:32.148068 25752 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.772s user 0.010s sys 0.031s
W20260807 11:10:32.187757 25752 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.812s user 0.011s sys 0.032s
W20260807 11:10:32.188128 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 824242us
I20260807 11:10:32.152030 28665 raft_consensus.cc:2726] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:32.156355 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
W20260807 11:10:32.195428 26987 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 381) took 1412 ms (client timeout 1653 ms). Trace:
W20260807 11:10:32.195878 26987 rpcz_store.cc:269] 0807 11:10:30.782743 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:30.794668 (+ 11925us) service_pool.cc:227] Handling call
0807 11:10:32.156173 (+1361505us) spinlock_profiling.cc:243] Waited 7.56 ms on lock 0x7b5800000f98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc46061b0 00007f3fc4605a94 00007f3fbdcedc04 00007f3fbdcedb92
0807 11:10:32.195359 (+ 39186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17399168}
I20260807 11:10:32.197778 26036 raft_consensus.cc:1261] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260807 11:10:32.198211 26036 pending_rounds.cc:76] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9: Aborting all ops after (but not including) 1
I20260807 11:10:32.198482 26036 pending_rounds.cc:97] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260807 11:10:32.198765 26036 raft_consensus.cc:2866] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:10:32.201905 28567 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:32.186055 26985 raft_consensus.cc:2454] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:32.076244 26037 raft_consensus.cc:2454] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:10:32.158434 25921 raft_consensus.cc:1261] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:10:32.154173 25796 rpcz_store.cc:276] 0807 11:10:30.014124 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:30.015097 (+ 973us) service_pool.cc:227] Handling call
0807 11:10:31.561058 (+1545961us) spinlock_profiling.cc:243] Waited 3.23 ms on lock 0x7b5800003c98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc461a2b0 00007f3fc46057f9 00007f3fc46052a2 00007f3fbdceb684
0807 11:10:31.739859 (+178801us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:31.744130 (+ 4271us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:32.153485 (+409355us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b54002f5150. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:32.153640 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":417966720}
I20260807 11:10:32.183081 28878 consensus_queue.cc:1046] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20260807 11:10:32.073967 25919 raft_consensus.cc:1226] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260807 11:10:32.207505 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:32.207506 25921 raft_consensus.cc:1261] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 11:10:32.212167 25919 raft_consensus.cc:1261] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260807 11:10:32.214007 25797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1553) took 1431 ms. Trace:
I20260807 11:10:32.215747 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:32.216862 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:32.206177 26938 leader_election.cc:302] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:32.207830 28567 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:32.184748 28088 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:32.214412 25797 rpcz_store.cc:276] 0807 11:10:30.782931 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:30.887511 (+104580us) service_pool.cc:227] Handling call
0807 11:10:30.887992 (+ 481us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:30.897489 (+ 9497us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:30.897533 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:31.069822 (+172289us) log.cc:842] Serialized 26 byte log entry
0807 11:10:31.331219 (+261397us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:10:31.331255 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:31.331320 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:32.129510 (+798190us) spinlock_profiling.cc:243] Waited 5.56 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:10:32.213689 (+ 84179us) raft_consensus.cc:1671] finished
0807 11:10:32.213719 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:32.213941 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9094,"raft.queue_time_us":101128,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":32285,"spinlock_wait_cycles":12798336,"thread_start_us":232696,"threads_started":1,"wal-append.queue_time_us":291812}
I20260807 11:10:32.159214 28616 raft_consensus.cc:504] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:32.241057 28724 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.055s user 0.021s sys 0.046s
I20260807 11:10:32.325847 28946 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:32.251610 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:32.220639 28871 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:32.226964 28738 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:32.236953 28633 raft_consensus.cc:2781] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:10:32.247679 26980 raft_consensus.cc:2379] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 6: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:32.357905 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:32.227226 28533 raft_consensus.cc:482] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:32.372177 26036 raft_consensus.cc:3037] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:32.377660 28674 raft_consensus.cc:3037] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:32.208225 28788 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:10:32.400450 26046 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.199s user 0.027s sys 0.033s
W20260807 11:10:32.406752 26046 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.208s user 0.027s sys 0.033s
I20260807 11:10:32.400694 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:32.365070 28724 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.182s user 0.022s sys 0.046s
W20260807 11:10:32.350451 28882 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.947s user 0.010s sys 0.028s
W20260807 11:10:32.418221 25986 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.040s user 0.040s sys 0.110s
W20260807 11:10:32.424686 25986 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.048s user 0.041s sys 0.110s
W20260807 11:10:32.351172 25931 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.088s user 0.020s sys 0.020s
W20260807 11:10:32.407346 28646 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.865s user 0.026s sys 0.011s
I20260807 11:10:32.453619 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:32.378790 28269 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:32.236784 28088 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:32.453051 28882 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.021s user 0.011s sys 0.029s
W20260807 11:10:32.455930 25931 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.200s user 0.020s sys 0.020s
W20260807 11:10:32.488324 28892 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.934s user 0.020s sys 0.018s
I20260807 11:10:32.411568 28457 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:32.505322 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:32.485459 28646 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.938s user 0.026s sys 0.011s
I20260807 11:10:32.420657 28633 raft_consensus.cc:686] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:32.515182 26035 raft_consensus.cc:1226] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
W20260807 11:10:32.517637 26936 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.974s user 0.037s sys 0.058s
I20260807 11:10:32.399348 28871 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:32.489918 28910 raft_consensus.cc:482] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:32.514480 28892 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.961s user 0.020s sys 0.019s
I20260807 11:10:32.488732 28778 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
W20260807 11:10:32.529296 26936 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.986s user 0.038s sys 0.058s
I20260807 11:10:32.413762 28533 raft_consensus.cc:504] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:32.411787 28738 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:32.490082 28797 raft_consensus.cc:482] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
W20260807 11:10:32.535512 25786 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.385s user 0.024s sys 0.033s
W20260807 11:10:32.540251 25750 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.921s user 0.019s sys 0.032s
I20260807 11:10:32.510957 28785 raft_consensus.cc:482] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
W20260807 11:10:32.553263 25750 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.934s user 0.019s sys 0.033s
W20260807 11:10:32.553640 25786 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.403s user 0.024s sys 0.033s
W20260807 11:10:32.555265 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 935969us
I20260807 11:10:32.493814 28269 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:32.542172 28789 leader_election.cc:288] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:10:32.413985 26975 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.025s user 0.017s sys 0.020s
I20260807 11:10:32.579344 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:32.580099 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:32.525171 28928 consensus_queue.cc:1035] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
W20260807 11:10:32.562944 27960 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.737s user 0.008s sys 0.015s
I20260807 11:10:32.595865 26036 raft_consensus.cc:2454] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:32.538223 28583 consensus_queue.cc:1046] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260807 11:10:32.612699 25920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1485) took 2634 ms. Trace:
W20260807 11:10:32.613623 26936 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.120.130:36465 timed out after 1.641s (SENT) [suppressed 49 similar messages]
I20260807 11:10:32.633085 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:32.631904 25920 rpcz_store.cc:276] 0807 11:10:29.978187 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:30.666147 (+687960us) service_pool.cc:227] Handling call
0807 11:10:30.673970 (+ 7823us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:30.679074 (+ 5104us) raft_consensus.cc:1520] Early marking committed up to index 5
0807 11:10:30.694617 (+ 15543us) log.cc:842] Serialized 14 byte log entry
0807 11:10:32.103232 (+1408615us) spinlock_profiling.cc:243] Waited 2.94 ms on lock 0x7b18002a2260. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbc77e345 00007f3fc55b5a39 00007f3fbc79b8bb 00007f3fbc799cbf 00007f3fbc79be22 00007f3fbf3455bb 00007f3fbf34b063 00007f3fbf34b57b 00007f3fbf6ce88a
0807 11:10:32.612216 (+508984us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:32.612287 (+ 71us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:32.612349 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:32.612453 (+ 104us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:32.612626 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4162,"prepare.queue_time_us":23928,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":38642,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":2149,"spinlock_wait_cycles":6766464,"thread_start_us":483676,"threads_started":1}
I20260807 11:10:32.541455 28457 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:32.648658 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:32.649955 25920 raft_consensus.cc:3037] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
W20260807 11:10:32.591795 26975 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.203s user 0.017s sys 0.028s
I20260807 11:10:32.662531 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
W20260807 11:10:32.630470 27960 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.806s user 0.008s sys 0.015s
I20260807 11:10:32.541347 28910 raft_consensus.cc:504] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:32.551268 28778 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:32.674369 25752 leader_election.cc:302] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:10:32.678695 25751 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.034s user 0.000s sys 0.033s
W20260807 11:10:32.701264 25751 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.084s user 0.000s sys 0.033s
I20260807 11:10:32.613188 28706 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:32.558707 28797 raft_consensus.cc:504] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:32.703051 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1597) took 2052 ms. Trace:
I20260807 11:10:32.559463 28785 raft_consensus.cc:504] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:32.711694 25869 leader_election.cc:302] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:32.712929 26034 rpcz_store.cc:276] 0807 11:10:30.650760 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:30.650933 (+ 173us) service_pool.cc:227] Handling call
0807 11:10:30.651230 (+ 297us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:31.039815 (+388585us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:32.702788 (+1662973us) spinlock_profiling.cc:243] Waited 388 ms on lock 0x7b5400312754. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:32.702975 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":891836800}
I20260807 11:10:32.725301 25869 leader_election.cc:302] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:32.694972 28864 log.cc:924] Time spent T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 2.370s user 0.004s sys 0.013s
I20260807 11:10:32.686710 28724 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:32.720558 28951 raft_consensus.cc:2781] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:32.712796 28873 raft_consensus.cc:482] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
W20260807 11:10:32.771185 25919 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.006s sys 0.028s
W20260807 11:10:32.772573 25919 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.546s user 0.006s sys 0.028s
W20260807 11:10:32.712881 26974 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.849s user 0.018s sys 0.041s
I20260807 11:10:32.722671 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:32.726296 28878 raft_consensus.cc:482] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:32.745756 28882 raft_consensus.cc:2781] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260807 11:10:32.796814 25755 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.911s user 0.016s sys 0.037s
I20260807 11:10:32.664515 28633 consensus_queue.cc:237] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:32.798240 26975 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 378) took 2007 ms. Trace:
I20260807 11:10:32.773316 28665 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:32.726158 28706 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:33.404680 25755 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.770s user 0.020s sys 0.046s
W20260807 11:10:33.181751 25921 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.701s user 0.007s sys 0.016s
W20260807 11:10:33.405508 25921 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.925s user 0.007s sys 0.024s
I20260807 11:10:32.807207 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1489) took 1208 ms. Trace:
W20260807 11:10:33.044420 25872 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.411s user 0.025s sys 0.042s
W20260807 11:10:33.408550 25872 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.775s user 0.031s sys 0.056s
I20260807 11:10:33.403956 28567 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:32.964788 25869 leader_election.cc:302] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:10:33.409754 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 1.78992e+06us
W20260807 11:10:33.410349 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 445672us
I20260807 11:10:33.410696 25872 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:32.837775 28873 raft_consensus.cc:504] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:32.991658 28533 leader_election.cc:288] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:33.413010 28533 raft_consensus.cc:2781] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:33.413532 28533 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:33.414060 28533 raft_consensus.cc:3037] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:33.414243 28873 leader_election.cc:288] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:33.429100 28873 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:32.749126 28722 raft_consensus.cc:2781] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:32.813097 28871 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:33.416038 28878 raft_consensus.cc:504] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:32.812108 28385 raft_consensus.cc:482] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:33.407502 25917 rpcz_store.cc:276] 0807 11:10:31.599276 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:31.978018 (+378742us) service_pool.cc:227] Handling call
0807 11:10:32.156566 (+178548us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:32.158581 (+ 2015us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:32.806946 (+648365us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b5400339fd0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:32.807134 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":403936000}
I20260807 11:10:33.035871 28738 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:33.039599 28948 raft_consensus.cc:482] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:33.417745 28665 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:32.772055 28088 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:32.820111 28951 raft_consensus.cc:686] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
W20260807 11:10:32.808055 28867 consensus_meta.cc:287] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.796s user 0.009s sys 0.005s
W20260807 11:10:33.121488 28957 log.cc:924] Time spent T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.179s user 0.000s sys 0.001s
W20260807 11:10:32.813591 26986 consensus_meta.cc:287] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.628s user 0.012s sys 0.000s
I20260807 11:10:33.502673 26986 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:33.503459 26986 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 355) took 1327 ms. Trace:
I20260807 11:10:32.771173 28724 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:33.340564 28759 raft_consensus.cc:3037] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
W20260807 11:10:32.777550 26974 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.915s user 0.020s sys 0.041s
I20260807 11:10:32.831725 28868 raft_consensus.cc:3032] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Stepping down as leader of term 2
W20260807 11:10:33.414958 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 608368us
W20260807 11:10:32.988322 25916 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.991s user 0.003s sys 0.012s
I20260807 11:10:32.800700 25920 raft_consensus.cc:2454] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
W20260807 11:10:33.510874 25916 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.514s user 0.005s sys 0.014s
I20260807 11:10:33.444262 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:33.511896 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:33.445619 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:33.513078 25808 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
W20260807 11:10:32.950003 26976 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.255s user 0.027s sys 0.039s
I20260807 11:10:33.446326 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:33.427173 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:33.121214 28778 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:10:32.807291 28782 consensus_meta.cc:287] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.639s user 0.010s sys 0.002s
W20260807 11:10:32.994333 28956 log.cc:924] Time spent T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.355s user 0.000s sys 0.001s
W20260807 11:10:33.341717 28959 log.cc:924] Time spent T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.660s user 0.001s sys 0.001s
W20260807 11:10:33.318817 28975 log.cc:924] Time spent T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
W20260807 11:10:33.020079 28850 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.419s user 0.027s sys 0.037s
I20260807 11:10:33.378443 28438 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:33.441668 25931 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.859s user 0.010s sys 0.033s
I20260807 11:10:33.481503 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:33.536485 25799 raft_consensus.cc:2454] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:10:32.803331 26979 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 380) took 2027 ms. Trace:
I20260807 11:10:33.493189 28385 raft_consensus.cc:504] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:33.523272 28871 consensus_queue.cc:1035] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20260807 11:10:33.513979 28868 raft_consensus.cc:729] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:33.502085 28867 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:33.561101 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:32.814616 28910 leader_election.cc:288] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:33.440163 28928 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
W20260807 11:10:33.441532 28619 consensus_meta.cc:287] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 1.285s user 0.005s sys 0.010s
I20260807 11:10:33.434774 28785 leader_election.cc:288] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:33.445693 28899 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:33.485067 28722 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:33.588280 28722 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:33.589085 28928 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:33.588471 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:33.591189 25799 raft_consensus.cc:2454] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:33.591913 28928 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:33.592206 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:33.506224 28449 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:33.499187 28948 raft_consensus.cc:504] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:33.506108 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:33.593451 25917 raft_consensus.cc:2454] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:33.593958 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1567) took 1002 ms. Trace:
I20260807 11:10:33.506548 28951 consensus_queue.cc:237] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:33.503947 26986 rpcz_store.cc:276] 0807 11:10:32.176356 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:32.176539 (+ 183us) service_pool.cc:227] Handling call
0807 11:10:33.503373 (+1326834us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:33.594571 25806 raft_consensus.cc:2454] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:33.486649 28616 leader_election.cc:288] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:33.512106 25800 raft_consensus.cc:2454] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:10:33.598932 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:33.450771 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:33.520087 26984 raft_consensus.cc:2454] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:33.600091 25796 raft_consensus.cc:2454] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:33.499981 28738 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:33.528705 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:33.499732 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:33.488947 28873 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:33.507786 28457 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:33.523495 26976 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.828s user 0.030s sys 0.051s
W20260807 11:10:33.526140 28892 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.941s user 0.001s sys 0.014s
I20260807 11:10:33.409529 28882 raft_consensus.cc:482] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:33.556752 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1607) took 2439 ms. Trace:
I20260807 11:10:33.592893 25799 raft_consensus.cc:2454] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
W20260807 11:10:33.538175 25931 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.975s user 0.010s sys 0.033s
W20260807 11:10:33.536453 28850 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.972s user 0.028s sys 0.059s
I20260807 11:10:33.524597 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:33.538868 28782 consensus_queue.cc:1035] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.078s
I20260807 11:10:33.597783 25808 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:10:33.633352 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:33.630623 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 273) took 2409 ms. Trace:
I20260807 11:10:33.634359 25808 raft_consensus.cc:2379] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 4: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:33.634260 26042 rpcz_store.cc:276] 0807 11:10:31.221163 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:31.221282 (+ 119us) service_pool.cc:227] Handling call
0807 11:10:32.036426 (+815144us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:32.040658 (+ 4232us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:33.630439 (+1589781us) spinlock_profiling.cc:243] Waited 766 ms on lock 0x7b54000bbb50. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:33.630564 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1762008448}
I20260807 11:10:33.635462 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:33.605584 25800 raft_consensus.cc:2379] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 3: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:33.536947 28088 raft_consensus.cc:2726] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260807 11:10:33.558236 28674 consensus_meta.cc:287] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 1.145s user 0.010s sys 0.002s
I20260807 11:10:33.596416 26974 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50960 (request call id 323) took 2528 ms. Trace:
W20260807 11:10:33.606105 28747 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.438s user 0.014s sys 0.036s
I20260807 11:10:33.538094 28641 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:33.647986 25919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1549) took 2035 ms. Trace:
W20260807 11:10:33.648075 28747 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.491s user 0.014s sys 0.037s
I20260807 11:10:33.564232 28856 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:33.608382 26936 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.975s user 0.010s sys 0.026s
I20260807 11:10:33.596539 28772 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:33.596868 28928 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:33.619275 26046 rpcz_store.cc:276] 0807 11:10:31.117242 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:31.117460 (+ 218us) service_pool.cc:227] Handling call
0807 11:10:31.117887 (+ 427us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:31.118307 (+ 420us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:31.118367 (+ 60us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:31.136027 (+ 17660us) log.cc:842] Serialized 26 byte log entry
0807 11:10:32.407313 (+1271286us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:10:32.407364 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:32.407444 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:33.556475 (+1149031us) raft_consensus.cc:1671] finished
0807 11:10:33.556503 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:33.556678 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1073,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":56163,"thread_start_us":1208311,"threads_started":1,"wal-append.queue_time_us":1562533}
I20260807 11:10:33.569375 28438 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:33.623472 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:33.660320 25796 raft_consensus.cc:2379] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 3: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:33.626788 25801 raft_consensus.cc:2454] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:33.594403 25917 rpcz_store.cc:276] 0807 11:10:32.594260 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:33.490889 (+896629us) service_pool.cc:227] Handling call
0807 11:10:33.593887 (+102998us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:33.630291 28724 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:33.538017 26979 rpcz_store.cc:276] 0807 11:10:30.775716 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:30.783691 (+ 7975us) service_pool.cc:227] Handling call
0807 11:10:30.915950 (+132259us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:31.038723 (+122773us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:32.802960 (+1764237us) spinlock_profiling.cc:243] Waited 24 ms on lock 0x7b5400760550. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:32.803228 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":55138560}
I20260807 11:10:33.593024 28449 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:33.631011 25918 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1488) took 2032 ms (client timeout 1791 ms). Trace:
I20260807 11:10:33.648275 28641 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:33.629293 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:33.669922 25799 raft_consensus.cc:2379] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 3: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
W20260807 11:10:33.616144 28892 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.056s user 0.001s sys 0.014s
I20260807 11:10:33.534323 28778 consensus_queue.cc:1035] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.089s
I20260807 11:10:33.608788 28738 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:33.646313 26974 rpcz_store.cc:276] 0807 11:10:31.065740 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:31.068947 (+ 3207us) service_pool.cc:227] Handling call
0807 11:10:31.081779 (+ 12832us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:31.082010 (+ 231us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 11:10:31.850413 (+768403us) log.cc:842] Serialized 14 byte log entry
0807 11:10:33.525756 (+1675343us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:33.529729 (+ 3973us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:10:33.535545 (+ 5816us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:33.555745 (+ 20200us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:33.593603 (+ 37858us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9323,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":7869,"thread_start_us":1649007,"threads_started":1,"wal-append.queue_time_us":1791504}
I20260807 11:10:33.648420 25919 rpcz_store.cc:276] 0807 11:10:31.612183 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:32.212914 (+600731us) service_pool.cc:227] Handling call
0807 11:10:32.213271 (+ 357us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:32.226339 (+ 13068us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:32.226370 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:32.226806 (+ 436us) log.cc:842] Serialized 26 byte log entry
0807 11:10:32.772980 (+546174us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:32.820272 (+ 47292us) log.cc:842] Serialized 14 byte log entry
0807 11:10:32.907959 (+ 87687us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:32.908073 (+ 114us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:33.647791 (+739718us) raft_consensus.cc:1671] finished
0807 11:10:33.647809 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:33.647930 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12672,"prepare.queue_time_us":43727,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":65,"raft.queue_time_us":367252,"raft.run_cpu_time_us":358,"raft.run_wall_time_us":85944,"thread_start_us":545877,"threads_started":1,"wal-append.queue_time_us":1384872}
I20260807 11:10:33.650835 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:33.652457 26936 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.019s user 0.010s sys 0.027s
I20260807 11:10:33.675060 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:33.623185 28882 raft_consensus.cc:3037] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:33.643682 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:33.696744 25808 raft_consensus.cc:2454] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:33.609431 28899 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:33.615921 28457 raft_consensus.cc:482] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:33.665387 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:33.666568 25918 rpcz_store.cc:269] 0807 11:10:31.598887 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:31.601187 (+ 2300us) service_pool.cc:227] Handling call
0807 11:10:33.630823 (+2029636us) spinlock_profiling.cc:243] Waited 54.2 ms on lock 0x7b54000d1954. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:33.630946 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":124569984}
I20260807 11:10:33.627903 28868 consensus_queue.cc:260] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:33.675508 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 1.06185e+06us
I20260807 11:10:33.675423 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:33.698113 25917 raft_consensus.cc:3037] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:33.696022 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:33.655380 28533 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:33.722457 25919 raft_consensus.cc:3037] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:33.710403 25800 raft_consensus.cc:2396] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 11:10:33.722147 28788 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:33.725934 28788 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:33.726753 28449 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:33.727694 28533 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:33.728066 28788 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:33.729800 28867 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:33.730545 28867 consensus_queue.cc:1046] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 11:10:33.732230 28385 leader_election.cc:288] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:33.701761 25931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 183) took 3675 ms. Trace:
I20260807 11:10:33.672427 25925 raft_consensus.cc:1261] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:10:33.733173 25931 rpcz_store.cc:276] 0807 11:10:30.021901 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:30.897564 (+875663us) service_pool.cc:227] Handling call
0807 11:10:31.075018 (+177454us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:31.095098 (+ 20080us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:31.098036 (+ 2938us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:31.193817 (+ 95781us) log.cc:842] Serialized 26 byte log entry
0807 11:10:32.524976 (+1331159us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:10:33.640364 (+1115388us) log.cc:842] Serialized 14 byte log entry
0807 11:10:33.656414 (+ 16050us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:33.663637 (+ 7223us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:33.667109 (+ 3472us) raft_consensus.cc:1671] finished
0807 11:10:33.667141 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:33.697490 (+ 30349us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":140069,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":6518,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":2651,"thread_start_us":2323557,"threads_started":2,"wal-append.queue_time_us":1518832}
I20260807 11:10:33.735461 26936 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:33.736061 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:33.736853 28754 raft_consensus.cc:482] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:33.737262 25875 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.012s sys 0.028s
W20260807 11:10:33.737810 25875 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.744s user 0.013s sys 0.029s
I20260807 11:10:33.737382 28754 raft_consensus.cc:504] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:33.722472 28503 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:33.738780 28503 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:33.719789 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:33.704214 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:33.409210 26975 rpcz_store.cc:276] 0807 11:10:30.771010 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:30.771211 (+ 201us) service_pool.cc:227] Handling call
0807 11:10:30.881562 (+110351us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:30.910989 (+ 29427us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:30.911039 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:30.969726 (+ 58687us) log.cc:842] Serialized 26 byte log entry
0807 11:10:31.358085 (+388359us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:10:31.922273 (+564188us) spinlock_profiling.cc:243] Waited 8.53 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:10:32.681515 (+759242us) log.cc:842] Serialized 14 byte log entry
0807 11:10:32.712966 (+ 31451us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:32.726739 (+ 13773us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:32.729414 (+ 2675us) raft_consensus.cc:1671] finished
0807 11:10:32.750223 (+ 20809us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:32.778578 (+ 28355us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1064,"prepare.queue_time_us":1305990,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":53,"raft.queue_time_us":20681,"raft.run_cpu_time_us":369,"raft.run_wall_time_us":38017,"spinlock_wait_cycles":19611904,"thread_start_us":1548179,"threads_started":2,"wal-append.queue_time_us":358110}
I20260807 11:10:33.744530 26936 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:33.746665 25918 raft_consensus.cc:3037] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:33.736406 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1486) took 3757 ms. Trace:
I20260807 11:10:33.659680 28772 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:33.750762 26936 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:33.740551 28754 leader_election.cc:288] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:33.697716 28665 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:33.666025 28928 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:33.699991 28747 raft_consensus.cc:482] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
W20260807 11:10:33.693549 28646 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.138s user 0.017s sys 0.044s
I20260807 11:10:33.760545 26936 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:33.763262 25919 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:33.762987 28503 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:33.778370 25919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1566) took 1189 ms. Trace:
I20260807 11:10:33.760948 28867 raft_consensus.cc:2726] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:33.781901 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:33.785481 25919 rpcz_store.cc:276] 0807 11:10:32.588881 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:33.675165 (+1086284us) service_pool.cc:227] Handling call
0807 11:10:33.778265 (+103100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:33.698984 28722 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:33.760222 28449 raft_consensus.cc:2781] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260807 11:10:33.782065 25803 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59118 (request call id 215) took 4551 ms (client timeout 1856 ms). Trace:
I20260807 11:10:33.794003 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:33.794247 25803 rpcz_store.cc:269] 0807 11:10:29.214905 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:29.216057 (+ 1152us) service_pool.cc:227] Handling call
0807 11:10:33.765837 (+4549780us) spinlock_profiling.cc:243] Waited 2.67 s on lock 0x7b54002f4c54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:33.765974 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6135812864}
I20260807 11:10:33.762054 28788 raft_consensus.cc:2726] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:33.794649 25919 raft_consensus.cc:2454] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:33.723212 28457 raft_consensus.cc:504] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:33.759219 25925 rpcz_store.cc:276] 0807 11:10:29.978771 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:30.665872 (+687101us) service_pool.cc:227] Handling call
0807 11:10:30.672452 (+ 6580us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:33.734629 (+3062177us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:33.736087 (+ 1458us) spinlock_profiling.cc:243] Waited 2.98 s on lock 0x7b5400185454. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:33.736318 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6848064896}
W20260807 11:10:33.775750 28968 log.cc:924] Time spent T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.076s user 0.000s sys 0.001s
I20260807 11:10:33.785516 28868 raft_consensus.cc:3037] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:33.797313 25919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1561) took 1239 ms. Trace:
I20260807 11:10:33.795468 26038 raft_consensus.cc:2454] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:33.805135 25920 raft_consensus.cc:2379] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 6: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:33.784824 28902 raft_consensus.cc:2726] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:33.797525 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:33.750131 25931 raft_consensus.cc:2454] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
W20260807 11:10:33.802866 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 392270us
I20260807 11:10:33.804527 25919 rpcz_store.cc:276] 0807 11:10:32.558178 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:33.793692 (+1235514us) service_pool.cc:227] Handling call
0807 11:10:33.797253 (+ 3561us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:10:33.783188 28646 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.235s user 0.018s sys 0.047s
I20260807 11:10:33.808566 25796 raft_consensus.cc:3037] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
W20260807 11:10:33.792105 25988 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.933s user 0.018s sys 0.025s
W20260807 11:10:33.808957 25786 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.165s user 0.016s sys 0.044s
I20260807 11:10:33.795667 25798 raft_consensus.cc:3037] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:33.798718 28449 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:33.812695 25917 raft_consensus.cc:2454] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:10:33.821693 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:33.820706 25918 raft_consensus.cc:2454] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:33.807893 28940 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:33.893097 28940 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:33.886845 28982 log.cc:924] Time spent T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.188s user 0.001s sys 0.001s
I20260807 11:10:33.897001 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1554) took 2283 ms. Trace:
I20260807 11:10:33.897617 25921 rpcz_store.cc:276] 0807 11:10:31.613819 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:32.213044 (+599225us) service_pool.cc:227] Handling call
0807 11:10:32.213328 (+ 284us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:32.237631 (+ 24303us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:32.237664 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:32.356891 (+119227us) log.cc:842] Serialized 26 byte log entry
0807 11:10:33.406106 (+1049215us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:33.406494 (+ 388us) log.cc:842] Serialized 14 byte log entry
0807 11:10:33.406675 (+ 181us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:33.406762 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:33.896708 (+489946us) raft_consensus.cc:1671] finished
0807 11:10:33.896735 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:33.896927 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13152,"prepare.queue_time_us":241073,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":19630,"raft.run_cpu_time_us":316,"raft.run_wall_time_us":9852,"spinlock_wait_cycles":14208,"thread_start_us":925419,"threads_started":1,"wal-append.queue_time_us":1229697}
I20260807 11:10:33.901468 28738 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:33.904093 28894 raft_consensus.cc:3037] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:33.904481 28269 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:33.905496 26976 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 386) took 2436 ms. Trace:
I20260807 11:10:33.905990 26976 rpcz_store.cc:276] 0807 11:10:31.469248 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:31.473229 (+ 3981us) service_pool.cc:227] Handling call
0807 11:10:31.540129 (+ 66900us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:10:31.615719 (+ 75590us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:31.618206 (+ 2487us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 11:10:31.694969 (+ 76763us) log.cc:842] Serialized 52 byte log entry
0807 11:10:33.623106 (+1928137us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:33.628211 (+ 5105us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:33.628296 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:33.905172 (+276876us) raft_consensus.cc:1671] finished
0807 11:10:33.905197 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:33.905404 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":125460,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":116,"thread_start_us":1918249,"threads_started":1,"wal-append.queue_time_us":1957623}
I20260807 11:10:33.910439 28899 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:33.910979 28772 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:33.912077 28928 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:33.913966 28457 leader_election.cc:288] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:33.810490 28873 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:33.885787 28449 raft_consensus.cc:3037] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:33.915452 28706 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:33.917316 25932 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1455) took 5378 ms (client timeout 1706 ms). Trace:
I20260807 11:10:33.917590 25926 raft_consensus.cc:1261] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260807 11:10:33.917837 25932 rpcz_store.cc:269] 0807 11:10:28.539101 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:28.542649 (+ 3548us) service_pool.cc:227] Handling call
0807 11:10:33.917054 (+5374405us) spinlock_profiling.cc:243] Waited 3.53 s on lock 0x7b54000357d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:33.917198 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8118325632}
I20260807 11:10:33.918958 25926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1484) took 4239 ms. Trace:
I20260807 11:10:33.919293 25926 rpcz_store.cc:276] 0807 11:10:29.679310 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:30.664933 (+985623us) service_pool.cc:227] Handling call
0807 11:10:33.917231 (+3252298us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:33.918012 (+ 781us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:33.918786 (+ 774us) spinlock_profiling.cc:243] Waited 3.25 s on lock 0x7b54000357d0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:33.918905 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7478138240}
I20260807 11:10:33.883769 28897 consensus_queue.cc:1046] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260807 11:10:33.926290 28106 raft_consensus.cc:3032] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:33.926690 28106 raft_consensus.cc:729] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:33.927659 28106 consensus_queue.cc:260] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:33.933928 26980 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 387) took 2464 ms (client timeout 2125 ms). Trace:
W20260807 11:10:33.934496 26980 rpcz_store.cc:269] 0807 11:10:31.469668 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:31.484153 (+ 14485us) service_pool.cc:227] Handling call
0807 11:10:33.933617 (+2449464us) spinlock_profiling.cc:243] Waited 658 ms on lock 0x7b54004c13d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:33.933830 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1513681280}
W20260807 11:10:33.937740 25655 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:40716 (request call id 245) took 12729 ms (client timeout 15000 ms). Trace:
I20260807 11:10:33.806843 28948 leader_election.cc:288] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:33.946568 28878 leader_election.cc:288] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:33.954916 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 1623) took 1451 ms. Trace:
I20260807 11:10:33.955513 26038 rpcz_store.cc:276] 0807 11:10:32.503020 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:32.504918 (+ 1898us) service_pool.cc:227] Handling call
0807 11:10:33.954692 (+1449774us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54000569d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:33.954839 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2932121856}
W20260807 11:10:33.959416 25806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1424) took 4819 ms (client timeout 1562 ms). Trace:
W20260807 11:10:33.959828 25806 rpcz_store.cc:269] 0807 11:10:29.139714 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:29.139867 (+ 153us) service_pool.cc:227] Handling call
0807 11:10:33.959219 (+4819352us) spinlock_profiling.cc:243] Waited 4.45 s on lock 0x7b54000483d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:33.959332 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10242590976}
I20260807 11:10:33.960662 28709 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:33.962363 25654 catalog_manager.cc:5495] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 reported cstate change: term changed from 2 to 3, leader changed from eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 3 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 11:10:33.965051 25925 raft_consensus.cc:2454] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:33.978904 28641 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:33.979758 25796 raft_consensus.cc:2454] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:10:33.982673 28873 raft_consensus.cc:482] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:33.983235 28873 raft_consensus.cc:504] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:33.985169 28873 leader_election.cc:288] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:33.982688 28940 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:33.993932 25920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35276 (request call id 193) took 1411 ms. Trace:
I20260807 11:10:33.994426 25920 rpcz_store.cc:276] 0807 11:10:32.586637 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:33.523160 (+936523us) service_pool.cc:227] Handling call
0807 11:10:33.993680 (+470520us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b540019ffd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:33.993856 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":465167104}
I20260807 11:10:33.995925 28797 leader_election.cc:288] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:33.997859 28780 raft_consensus.cc:3037] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:33.998293 28449 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:34.012567 25981 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.039s sys 0.031s
W20260807 11:10:34.013873 25981 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.502s user 0.039s sys 0.031s
I20260807 11:10:33.826062 25931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35276 (request call id 192) took 1263 ms. Trace:
I20260807 11:10:33.820935 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:33.823024 25988 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.964s user 0.019s sys 0.025s
I20260807 11:10:33.821821 28438 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:33.826305 25786 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.182s user 0.017s sys 0.044s
I20260807 11:10:34.042611 25786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53324 (request call id 1443) took 2975 ms. Trace:
I20260807 11:10:34.042831 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:33.781806 28747 raft_consensus.cc:504] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:33.885680 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 64227.7us
I20260807 11:10:33.886098 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35276 (request call id 191) took 1335 ms. Trace:
I20260807 11:10:34.045912 25917 rpcz_store.cc:276] 0807 11:10:32.550385 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:33.661294 (+1110909us) service_pool.cc:227] Handling call
0807 11:10:33.886010 (+224716us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:34.046950 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:33.938261 25655 rpcz_store.cc:269] 0807 11:10:21.208918 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:21.211524 (+ 2606us) service_pool.cc:227] Handling call
0807 11:10:33.937632 (+12726108us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 11:10:28.178259 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 11:10:28.360527 (+182268us) write_op.cc:432] Acquiring schema lock in shared mode
0807 11:10:28.368311 (+ 7784us) write_op.cc:435] Acquired schema lock
0807 11:10:28.369762 (+ 1451us) tablet.cc:660] Decoding operations
0807 11:10:28.521493 (+151731us) write_op.cc:620] Acquiring the partition lock for write op
0807 11:10:28.551821 (+ 30328us) write_op.cc:641] Partition lock acquired for write op
0807 11:10:28.554592 (+ 2771us) tablet.cc:683] Acquiring locks for 5 operations
0807 11:10:28.626003 (+ 71411us) tablet.cc:699] Row locks acquired
0807 11:10:28.642671 (+ 16668us) write_op.cc:260] PREPARE: finished
0807 11:10:28.685665 (+ 42994us) write_op.cc:270] Start()
0807 11:10:28.841604 (+155939us) write_op.cc:276] Timestamp: P: 1786101028678178 usec, L: 0
0807 11:10:28.847843 (+ 6239us) op_driver.cc:331] REPLICATION: starting
0807 11:10:28.975377 (+127534us) log.cc:842] Serialized 2085 byte log entry
0807 11:10:29.542795 (+567418us) op_driver.cc:444] REPLICATION: finished
0807 11:10:29.645460 (+102665us) write_op.cc:301] APPLY: starting
0807 11:10:29.645586 (+ 126us) tablet.cc:1365] starting BulkCheckPresence
0807 11:10:29.645830 (+ 244us) tablet.cc:1368] finished BulkCheckPresence
0807 11:10:29.646022 (+ 192us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 11:10:29.647145 (+ 1123us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 11:10:29.647453 (+ 308us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0807 11:10:29.651245 (+ 3792us) write_op.cc:312] APPLY: finished
0807 11:10:29.724500 (+ 73255us) log.cc:842] Serialized 47 byte log entry
0807 11:10:29.733718 (+ 9218us) write_op.cc:489] Releasing partition, row and schema locks
0807 11:10:29.761688 (+ 27970us) write_op.cc:454] Released schema lock
0807 11:10:29.762529 (+ 841us) write_op.cc:341] FINISH: Updating metrics
0807 11:10:29.881999 (+119470us) write_op.cc:489] Releasing partition, row and schema locks
0807 11:10:29.893048 (+ 11049us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":20403,"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":102169,"apply.run_cpu_time_us":4917,"apply.run_wall_time_us":264942,"num_ops":5,"prepare.queue_time_us":273814,"prepare.run_cpu_time_us":30482,"prepare.run_wall_time_us":898615,"raft.queue_time_us":184967,"raft.run_cpu_time_us":31921,"raft.run_wall_time_us":136931,"replication_time_us":694794,"thread_start_us":433692,"threads_started":4,"wal-append.queue_time_us":230912,"wal-append.run_cpu_time_us":33151,"wal-append.run_wall_time_us":1679234}]]}
I20260807 11:10:33.886538 25918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1564) took 1324 ms. Trace:
W20260807 11:10:33.888441 28991 log.cc:924] Time spent T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.116s user 0.000s sys 0.000s
W20260807 11:10:33.888826 25983 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.080s user 0.000s sys 0.046s
I20260807 11:10:34.053555 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:34.054102 25983 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.358s user 0.000s sys 0.047s
I20260807 11:10:33.889173 28722 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:34.015241 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 503628us
I20260807 11:10:34.055413 25981 leader_election.cc:302] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:34.057703 25919 raft_consensus.cc:2454] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
W20260807 11:10:34.032115 25755 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.573s user 0.005s sys 0.006s
I20260807 11:10:34.040493 25932 raft_consensus.cc:1261] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:10:34.059067 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1558) took 2444 ms. Trace:
I20260807 11:10:34.059513 25932 rpcz_store.cc:276] 0807 11:10:31.615098 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:33.918901 (+2303803us) service_pool.cc:227] Handling call
0807 11:10:34.012176 (+ 93275us) spinlock_profiling.cc:243] Waited 92.4 ms on lock 0x7b58002e0e98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc461a2b0 00007f3fc46057f9 00007f3fc46052a2 00007f3fbdceb684
0807 11:10:34.012445 (+ 269us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:34.058741 (+ 46296us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:34.058998 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":212447104}
I20260807 11:10:33.965763 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1562) took 1405 ms. Trace:
W20260807 11:10:34.075831 25916 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.034s sys 0.058s
W20260807 11:10:34.076236 25916 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.543s user 0.034s sys 0.058s
I20260807 11:10:34.025207 25926 raft_consensus.cc:1226] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:34.043803 25920 raft_consensus.cc:2379] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 4: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
W20260807 11:10:34.044108 25750 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.000s sys 0.013s
I20260807 11:10:34.048202 25918 rpcz_store.cc:276] 0807 11:10:32.562295 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:33.703861 (+1141566us) service_pool.cc:227] Handling call
0807 11:10:33.886470 (+182609us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:10:34.077190 25750 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.615s user 0.000s sys 0.013s
I20260807 11:10:34.043164 25786 rpcz_store.cc:276] 0807 11:10:31.067305 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:31.110773 (+ 43468us) service_pool.cc:227] Handling call
0807 11:10:31.111152 (+ 379us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:32.554344 (+1443192us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:32.554379 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:32.555954 (+ 1575us) log.cc:842] Serialized 26 byte log entry
0807 11:10:34.042057 (+1486103us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:34.042128 (+ 71us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:34.042239 (+ 111us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:34.042301 (+ 62us) raft_consensus.cc:1671] finished
0807 11:10:34.042320 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:34.042507 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":74449,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":7368,"thread_start_us":2801263,"threads_started":2,"wal-append.queue_time_us":1344803}
W20260807 11:10:34.068395 25755 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.610s user 0.006s sys 0.006s
W20260807 11:10:34.071683 25869 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.007s sys 0.011s
I20260807 11:10:34.183140 25752 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:34.106473 25750 leader_election.cc:302] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:34.082476 25981 leader_election.cc:398] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Vote denied by peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465) with higher term. Message: Invalid argument: T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5.
I20260807 11:10:34.078228 25918 raft_consensus.cc:1226] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:34.172013 25917 raft_consensus.cc:1261] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260807 11:10:34.208031 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 147113us
W20260807 11:10:34.208904 25869 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.747s user 0.014s sys 0.018s
I20260807 11:10:34.044085 28911 raft_consensus.cc:482] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
W20260807 11:10:34.211336 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 104975us
I20260807 11:10:34.131768 25926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1559) took 2516 ms. Trace:
I20260807 11:10:34.182215 28498 consensus_queue.cc:1046] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260807 11:10:34.081578 25916 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1539) took 2480 ms. Trace:
I20260807 11:10:34.214004 28293 raft_consensus.cc:3037] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:34.214532 28498 raft_consensus.cc:3032] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:34.148931 28934 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:34.214964 28498 raft_consensus.cc:729] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:34.215389 28934 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:34.208240 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 188) took 2591 ms. Trace:
I20260807 11:10:34.213510 28964 raft_consensus.cc:2781] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:34.213464 28647 raft_consensus.cc:482] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:34.216681 28964 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:34.216065 28498 consensus_queue.cc:260] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:34.217206 28964 raft_consensus.cc:3037] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:34.216833 28647 raft_consensus.cc:504] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:34.192925 28902 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:34.190471 28747 leader_election.cc:288] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:34.177235 28449 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195)
I20260807 11:10:34.192456 28882 raft_consensus.cc:504] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:34.218585 28106 raft_consensus.cc:3037] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
W20260807 11:10:34.189716 29003 log.cc:924] Time spent T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
I20260807 11:10:34.216526 25917 rpcz_store.cc:276] 0807 11:10:31.616663 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:34.082245 (+2465582us) service_pool.cc:227] Handling call
0807 11:10:34.171550 (+ 89305us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:34.207964 (+ 36414us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:34.208179 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:34.207046 25981 leader_election.cc:401] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260807 11:10:34.025436 25931 rpcz_store.cc:276] 0807 11:10:32.563257 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:33.735693 (+1172436us) service_pool.cc:227] Handling call
0807 11:10:33.825955 (+ 90262us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:34.148638 28966 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:34.232112 28966 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:34.232798 28462 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:34.214304 25916 rpcz_store.cc:276] 0807 11:10:31.601211 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:31.978283 (+377072us) service_pool.cc:227] Handling call
0807 11:10:31.978692 (+ 409us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:31.993357 (+ 14665us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:31.993385 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:31.996465 (+ 3080us) log.cc:842] Serialized 26 byte log entry
0807 11:10:33.511284 (+1514819us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:34.076653 (+565369us) log.cc:842] Serialized 14 byte log entry
0807 11:10:34.081161 (+ 4508us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:34.081270 (+ 109us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:34.081315 (+ 45us) raft_consensus.cc:1671] finished
0807 11:10:34.081332 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:34.081501 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":87,"prepare.queue_time_us":136135,"prepare.run_cpu_time_us":23,"prepare.run_wall_time_us":24,"raft.queue_time_us":14136,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":65913,"thread_start_us":2057858,"threads_started":2,"wal-append.queue_time_us":1647590}
I20260807 11:10:34.233426 29001 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:34.233410 28462 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:34.234102 29001 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:34.235777 28462 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:34.236272 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 454457us
I20260807 11:10:34.212447 25926 rpcz_store.cc:276] 0807 11:10:31.615404 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:33.920149 (+2304745us) service_pool.cc:227] Handling call
0807 11:10:34.008314 (+ 88165us) spinlock_profiling.cc:243] Waited 85.6 ms on lock 0x7b58002e0e98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc461a2b0 00007f3fc46057f9 00007f3fc46052a2 00007f3fbdceb684
0807 11:10:34.008648 (+ 334us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:34.101364 (+ 92716us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:34.131662 (+ 30298us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":196782976}
I20260807 11:10:34.236791 28966 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:34.219336 28498 raft_consensus.cc:3037] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:34.212844 25750 leader_election.cc:302] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:34.230113 26046 raft_consensus.cc:1226] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:34.180946 28567 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:34.238391 28567 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:34.071426 28871 raft_consensus.cc:2726] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 11:10:34.236347 28462 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:34.243238 26936 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:10:34.212746 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 751147us
I20260807 11:10:34.245453 28449 raft_consensus.cc:2781] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:34.245836 28449 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:34.236654 29001 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:34.246181 28449 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:34.106734 28761 raft_consensus.cc:482] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:34.243147 28462 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:34.210980 28780 raft_consensus.cc:482] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:34.082449 28665 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:34.247825 26936 leader_election.cc:302] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:10:34.254166 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 197434us
I20260807 11:10:34.259873 25981 leader_election.cc:302] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:34.261772 25869 leader_election.cc:302] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:34.265749 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:34.266676 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 53916.8us
I20260807 11:10:34.266786 26038 raft_consensus.cc:2454] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:10:34.275524 25750 leader_election.cc:398] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Vote denied by peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465) with higher term. Message: Invalid argument: T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 2. Current term is 3.
I20260807 11:10:34.227042 28946 raft_consensus.cc:482] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:34.338425 25750 leader_election.cc:302] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:34.341292 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:34.263484 28894 raft_consensus.cc:2781] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:10:34.219931 28902 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:34.209084 28423 consensus_queue.cc:1035] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260807 11:10:34.343834 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:34.344355 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:34.344996 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:34.368559 26034 raft_consensus.cc:3037] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:34.369041 26041 raft_consensus.cc:2454] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:34.370293 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:34.371073 26041 raft_consensus.cc:2454] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:34.372035 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:34.345296 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:34.225060 28747 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
W20260807 11:10:34.346818 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 87021.1us
W20260807 11:10:34.342809 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 78455.9us
I20260807 11:10:34.073760 25925 rpcz_store.cc:276] 0807 11:10:32.560168 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:33.807725 (+1247557us) service_pool.cc:227] Handling call
0807 11:10:33.965682 (+157957us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:34.346698 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:34.349864 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 88190.4us
I20260807 11:10:34.350701 28785 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:34.226900 28244 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:34.261615 28567 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:34.361305 26038 raft_consensus.cc:2396] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 11:10:34.362205 26039 raft_consensus.cc:1226] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:34.224273 28911 raft_consensus.cc:504] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:34.264024 29006 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:34.338892 28772 raft_consensus.cc:2726] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:34.259038 28761 raft_consensus.cc:504] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:34.347631 28385 raft_consensus.cc:2726] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465) with higher term. Message: Invalid argument: T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5.
I20260807 11:10:34.358639 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:34.259995 28780 raft_consensus.cc:504] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:34.264397 28665 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:34.366622 25750 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:34.376129 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:34.713097 26936 leader_election.cc:302] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:34.713527 26041 raft_consensus.cc:2454] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:34.411293 26034 raft_consensus.cc:2454] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:34.715595 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:34.716450 25926 raft_consensus.cc:2362] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5. Re-sending same reply.
I20260807 11:10:34.720268 25926 raft_consensus.cc:1261] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260807 11:10:34.722602 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:34.722976 25981 leader_election.cc:302] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:34.723419 25786 raft_consensus.cc:2379] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 3: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:34.725171 25981 leader_election.cc:398] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Vote denied by peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465) with higher term. Message: Invalid argument: T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4.
I20260807 11:10:34.725637 25981 leader_election.cc:401] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260807 11:10:34.726645 25869 leader_election.cc:302] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:34.741921 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:34.749235 28423 raft_consensus.cc:2781] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:34.749483 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:34.750029 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:34.750304 25916 raft_consensus.cc:3037] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:34.750195 28873 raft_consensus.cc:2781] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:34.748466 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:34.752311 25926 raft_consensus.cc:2379] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 3: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:34.752496 25786 raft_consensus.cc:3037] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:34.751380 28438 raft_consensus.cc:2781] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:34.755102 26036 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:10:34.755820 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:34.756575 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:34.757598 26038 raft_consensus.cc:2396] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 11:10:34.758410 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:34.758991 26038 raft_consensus.cc:2454] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:34.761394 26038 raft_consensus.cc:1261] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260807 11:10:34.762444 25869 leader_election.cc:302] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
W20260807 11:10:34.408838 26939 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.765s user 0.000s sys 0.015s
I20260807 11:10:34.763981 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:34.767221 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:34.354771 28797 raft_consensus.cc:2781] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:34.770072 29010 raft_consensus.cc:482] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:34.791924 29001 raft_consensus.cc:2781] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:34.792457 29001 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:34.792371 29010 raft_consensus.cc:504] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:34.793020 29001 raft_consensus.cc:3037] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:34.795701 28882 leader_election.cc:288] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:34.412798 28293 raft_consensus.cc:2726] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:34.798134 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:34.798821 26041 raft_consensus.cc:2454] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:10:34.802708 25869 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
W20260807 11:10:34.768214 26939 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.303s user 0.004s sys 0.019s
I20260807 11:10:34.768659 25926 raft_consensus.cc:2454] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:10:34.374552 28928 raft_consensus.cc:2726] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 11:10:34.404852 28934 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:34.403993 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:34.839872 26035 raft_consensus.cc:2454] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:10:34.810812 26033 raft_consensus.cc:2396] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 11:10:34.354588 28940 raft_consensus.cc:2781] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:34.812662 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:34.817588 26037 raft_consensus.cc:3037] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 4
W20260807 11:10:34.818396 25919 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.005s sys 0.004s
W20260807 11:10:34.819345 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 452816us
I20260807 11:10:34.359599 28871 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:34.384706 28647 leader_election.cc:288] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:34.406690 28894 raft_consensus.cc:686] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
W20260807 11:10:34.620484 29022 log.cc:924] Time spent T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.114s user 0.000s sys 0.001s
I20260807 11:10:34.782202 28385 raft_consensus.cc:2726] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465) with higher term. Message: Invalid argument: T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4.
I20260807 11:10:34.830674 25786 raft_consensus.cc:2454] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:10:34.834782 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:34.835619 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:34.836680 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:34.837509 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:34.841171 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:34.842674 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:34.749984 28423 raft_consensus.cc:686] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:34.762737 28438 raft_consensus.cc:482] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:34.363390 25931 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a"
dest_uuid: "4fd6381544574ddca559ce63b9243efc"
mode: GRACEFUL
new_leader_uuid: "b22d69db45194908ad3d345eba9badd9"
from {username='slave'} at 127.0.0.1:34992
I20260807 11:10:34.887882 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:34.426963 28946 raft_consensus.cc:504] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:34.804854 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:34.939950 25801 raft_consensus.cc:2396] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 11:10:34.900588 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:34.940701 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:34.941705 26035 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 1.
I20260807 11:10:34.939546 26033 raft_consensus.cc:2454] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
W20260807 11:10:34.940884 25919 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.682s user 0.007s sys 0.005s
I20260807 11:10:34.941870 25750 leader_election.cc:302] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:34.890812 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:34.402909 28462 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:34.519285 28902 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:34.943503 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:34.418193 28747 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:34.882355 25869 leader_election.cc:302] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:34.955049 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:34.940721 25803 raft_consensus.cc:2379] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 3: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:34.944919 25786 raft_consensus.cc:2454] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:34.839354 29010 leader_election.cc:288] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:34.805619 28797 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:34.705605 28964 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:34.755553 28244 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:34.890538 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:34.844401 28761 leader_election.cc:288] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:34.762984 29006 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:34.845956 28785 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:34.950932 25750 leader_election.cc:302] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:35.467674 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:34.973235 25799 raft_consensus.cc:2379] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 6: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:34.952642 25796 raft_consensus.cc:2454] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:35.472061 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:35.013429 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:34.411726 25798 raft_consensus.cc:2454] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:35.474203 25797 raft_consensus.cc:2454] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:35.001760 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:35.477650 25798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1573) took 1988 ms. Trace:
I20260807 11:10:35.479122 25801 raft_consensus.cc:2454] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:35.478490 25800 raft_consensus.cc:3037] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 3
I20260807 11:10:34.951892 25808 raft_consensus.cc:2396] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260807 11:10:35.119680 25916 raft_consensus.cc:2454] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:10:34.970728 25931 raft_consensus.cc:595] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Received request to transfer leadership to TS b22d69db45194908ad3d345eba9badd9
I20260807 11:10:35.045439 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1555) took 3431 ms. Trace:
I20260807 11:10:34.944696 25919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1563) took 2383 ms. Trace:
I20260807 11:10:35.479229 25798 rpcz_store.cc:276] 0807 11:10:33.488952 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:33.653861 (+164909us) service_pool.cc:227] Handling call
0807 11:10:35.477218 (+1823357us) spinlock_profiling.cc:243] Waited 43.1 ms on lock 0x7b54002f5654. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:35.477439 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":99037440}
I20260807 11:10:35.486116 25921 rpcz_store.cc:276] 0807 11:10:31.614117 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:33.900403 (+2286286us) service_pool.cc:227] Handling call
0807 11:10:34.008943 (+108540us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:34.010156 (+ 1213us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:34.010206 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:34.010817 (+ 611us) log.cc:842] Serialized 26 byte log entry
0807 11:10:34.338151 (+327334us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:34.361807 (+ 23656us) log.cc:842] Serialized 14 byte log entry
0807 11:10:34.377947 (+ 16140us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:34.378043 (+ 96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:35.045148 (+667105us) raft_consensus.cc:1671] finished
0807 11:10:35.045180 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:35.045353 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23027,"prepare.queue_time_us":26431,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":56,"raft.queue_time_us":22188,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":21044,"thread_start_us":267889,"threads_started":1,"wal-append.queue_time_us":478623}
I20260807 11:10:35.486935 25919 rpcz_store.cc:276] 0807 11:10:32.561157 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:34.058645 (+1497488us) service_pool.cc:227] Handling call
0807 11:10:34.149051 (+ 90406us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:34.221206 (+ 72155us) raft_consensus.cc:1520] Early marking committed up to index 5
0807 11:10:34.221254 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:34.226067 (+ 4813us) log.cc:842] Serialized 26 byte log entry
0807 11:10:34.944373 (+718306us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:34.944411 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:34.944482 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:34.944523 (+ 41us) raft_consensus.cc:1671] finished
0807 11:10:34.944534 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:34.944645 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3515,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":105,"thread_start_us":685211,"threads_started":1,"wal-append.queue_time_us":604456}
I20260807 11:10:35.486670 25931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34992 (request call id 10) took 1224 ms. Trace:
I20260807 11:10:35.497741 25931 rpcz_store.cc:276] 0807 11:10:34.262165 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:34.264841 (+ 2676us) service_pool.cc:227] Handling call
0807 11:10:35.486376 (+1221535us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:10:35.290179 25981 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.030s sys 0.055s
W20260807 11:10:35.501745 25981 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.756s user 0.038s sys 0.059s
I20260807 11:10:35.076632 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:35.021478 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:35.519668 25660 catalog_manager.cc:5495] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:35.522559 26974 raft_consensus.cc:1261] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260807 11:10:34.943619 28293 raft_consensus.cc:2741] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
W20260807 11:10:35.293489 26042 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.876s user 0.041s sys 0.059s
W20260807 11:10:35.525043 26936 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.679s user 0.003s sys 0.007s
I20260807 11:10:35.525261 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
W20260807 11:10:35.525692 26936 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.687s user 0.003s sys 0.008s
I20260807 11:10:35.525779 28761 raft_consensus.cc:2781] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:35.526748 28761 raft_consensus.cc:686] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
W20260807 11:10:35.529060 26938 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.683s user 0.002s sys 0.009s
I20260807 11:10:35.085837 28894 consensus_queue.cc:237] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:35.186383 28462 raft_consensus.cc:482] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:34.959746 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:35.531109 26033 raft_consensus.cc:2454] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:35.531030 28462 raft_consensus.cc:504] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:35.531802 28747 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:35.287544 28873 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.033s sys 0.060s
W20260807 11:10:35.533213 28873 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.781s user 0.039s sys 0.081s
I20260807 11:10:35.533738 28873 raft_consensus.cc:686] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:35.534597 26980 raft_consensus.cc:2454] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:35.067739 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:34.948832 28940 raft_consensus.cc:686] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:35.538580 28940 consensus_queue.cc:237] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:35.539160 29006 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:35.099710 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
W20260807 11:10:35.306530 29025 log.cc:924] Time spent T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.325s user 0.000s sys 0.002s
I20260807 11:10:34.942752 28438 raft_consensus.cc:3037] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:35.541606 29034 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:35.103117 29016 log.cc:924] Time spent T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.653s user 0.005s sys 0.007s
I20260807 11:10:35.542528 29034 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:34.973695 28665 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:35.158046 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:35.012805 28423 consensus_queue.cc:237] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:35.127435 28106 consensus_meta.cc:287] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.898s user 0.012s sys 0.000s
I20260807 11:10:35.126566 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:35.326371 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:34.891073 28871 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:35.306819 28722 raft_consensus.cc:482] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:34.957561 28647 raft_consensus.cc:2726] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 11:10:35.525437 26042 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.141s user 0.045s sys 0.066s
W20260807 11:10:35.329988 29001 consensus_meta.cc:287] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.536s user 0.011s sys 0.000s
I20260807 11:10:35.392884 28911 leader_election.cc:288] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:34.970125 28780 leader_election.cc:288] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:10:35.529790 26938 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.684s user 0.002s sys 0.009s
I20260807 11:10:35.270912 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:10:35.110386 28449 consensus_meta.cc:287] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.852s user 0.007s sys 0.003s
I20260807 11:10:35.535558 28873 consensus_queue.cc:237] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:35.529150 26034 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.554s user 0.000s sys 0.011s
I20260807 11:10:35.546372 26987 raft_consensus.cc:2379] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 3: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:35.549156 26985 raft_consensus.cc:2454] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:35.544713 28244 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:35.584863 26034 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.630s user 0.000s sys 0.012s
I20260807 11:10:35.027350 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:35.580312 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:34.828421 28882 raft_consensus.cc:2781] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:35.593621 28882 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:35.594275 28882 raft_consensus.cc:3037] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:35.581528 28449 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:35.504879 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 779803us
I20260807 11:10:35.598532 28449 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:35.598737 28747 consensus_queue.cc:1035] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:10:35.599066 25981 leader_election.cc:302] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:35.600497 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 718058us
I20260807 11:10:35.600821 29031 consensus_queue.cc:1035] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:10:35.603020 29031 raft_consensus.cc:2726] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:35.203318 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:35.582146 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 631066us
W20260807 11:10:35.557637 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 711999us
I20260807 11:10:35.264853 26037 raft_consensus.cc:2454] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:10:35.300604 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:35.605280 25786 raft_consensus.cc:2379] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 3: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:35.535792 28964 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:35.545341 28761 consensus_queue.cc:237] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:35.555397 29001 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:35.592669 28797 raft_consensus.cc:3037] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:35.546061 28665 raft_consensus.cc:2781] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:35.578514 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:35.619969 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43246 (request call id 298) took 1270 ms. Trace:
I20260807 11:10:35.625010 26037 rpcz_store.cc:276] 0807 11:10:34.349415 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:34.358389 (+ 8974us) service_pool.cc:227] Handling call
0807 11:10:35.619736 (+1261347us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:35.626977 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:35.627162 28462 leader_election.cc:288] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:35.628027 26037 raft_consensus.cc:3037] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
W20260807 11:10:35.349761 29011 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.018s sys 0.073s
W20260807 11:10:35.633867 29011 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.882s user 0.018s sys 0.074s
I20260807 11:10:35.621333 26036 raft_consensus.cc:2396] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 11:10:35.259087 28946 leader_election.cc:288] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:35.637672 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:35.638233 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:35.639331 29031 consensus_queue.cc:1035] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 11:10:35.640228 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:35.557987 29034 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:35.641482 25750 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:35.621876 26986 raft_consensus.cc:2379] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 6: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:35.620438 26039 raft_consensus.cc:2454] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 1.
I20260807 11:10:35.652823 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:35.654453 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:35.659000 29052 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:35.659909 29052 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:35.660902 28785 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:35.632823 28871 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:35.662820 28438 raft_consensus.cc:504] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:35.667425 25750 leader_election.cc:302] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:35.669052 28244 raft_consensus.cc:2726] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:35.669782 28244 raft_consensus.cc:2726] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:35.670882 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 295) took 1411 ms. Trace:
I20260807 11:10:35.671408 26046 rpcz_store.cc:276] 0807 11:10:34.259478 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:34.260922 (+ 1444us) service_pool.cc:227] Handling call
0807 11:10:34.261177 (+ 255us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:35.577711 (+1316534us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:35.577774 (+ 63us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:35.577851 (+ 77us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:10:35.577918 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:35.670369 (+ 92451us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b540038de54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:35.670424 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:35.670643 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2982694784}
I20260807 11:10:35.655880 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:35.624735 28665 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:35.701644 26035 raft_consensus.cc:2379] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 4: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:35.638852 25798 raft_consensus.cc:2454] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:35.705149 26036 raft_consensus.cc:2454] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:35.708536 25800 raft_consensus.cc:2454] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:35.552383 28722 raft_consensus.cc:504] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:35.709190 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 106337us
I20260807 11:10:35.704635 28665 raft_consensus.cc:3037] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
W20260807 11:10:35.715916 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 48716.6us
W20260807 11:10:35.716351 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 125882us
I20260807 11:10:35.717337 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:35.718971 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:35.719866 26035 raft_consensus.cc:2379] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 4: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:35.720970 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:35.724447 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:35.725651 25931 raft_consensus.cc:2379] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 4: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:35.727005 26938 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:35.727352 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:35.728132 25919 raft_consensus.cc:3037] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:35.728289 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:35.729142 25786 raft_consensus.cc:2454] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:35.729935 25750 leader_election.cc:302] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:35.730906 25806 raft_consensus.cc:1226] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260807 11:10:35.734745 25660 catalog_manager.cc:5495] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: term changed from 1 to 3, leader changed from b22d69db45194908ad3d345eba9badd9 (127.23.120.131) to eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132). New cstate: current_term: 3 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:10:35.737219 28449 raft_consensus.cc:2781] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:35.737725 28449 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:35.738152 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:10:35.738226 28449 raft_consensus.cc:3037] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:35.739197 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:10:35.739560 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:35.741933 25801 raft_consensus.cc:1261] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:10:35.742661 26938 leader_election.cc:302] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:35.743844 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1537) took 1401 ms. Trace:
I20260807 11:10:35.716162 26979 raft_consensus.cc:2454] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:35.719377 26936 leader_election.cc:302] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:35.755165 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:35.756062 26038 raft_consensus.cc:2454] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:35.732349 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:35.767619 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:35.768262 26036 raft_consensus.cc:2454] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:35.769307 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:35.769182 28946 consensus_queue.cc:1035] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:35.770579 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:35.770654 29052 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:35.771268 26036 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:35.772151 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:35.773855 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:35.774587 26046 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:35.775080 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:35.775413 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:35.776067 26041 raft_consensus.cc:2379] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 3: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:35.732319 28797 raft_consensus.cc:504] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:35.778847 28797 leader_election.cc:288] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:35.779574 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:35.780936 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 49632.7us
I20260807 11:10:35.781435 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:35.782258 28244 raft_consensus.cc:2781] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:35.780517 26046 raft_consensus.cc:2454] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:35.782944 28244 raft_consensus.cc:686] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:35.783308 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:35.784749 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:35.785074 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:35.785544 26046 raft_consensus.cc:2454] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:35.782843 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:35.775439 25919 raft_consensus.cc:2454] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:35.790118 26036 raft_consensus.cc:2454] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:35.795307 28503 raft_consensus.cc:2781] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:35.795928 28503 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:35.796355 28503 raft_consensus.cc:3037] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:35.797901 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:35.798399 28722 leader_election.cc:288] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:35.799782 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 57212.3us
I20260807 11:10:35.762562 28438 leader_election.cc:288] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:35.802788 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 83555.7us
I20260807 11:10:35.746309 26042 rpcz_store.cc:276] 0807 11:10:34.342430 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:34.342597 (+ 167us) service_pool.cc:227] Handling call
0807 11:10:34.361780 (+ 19183us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:34.361999 (+ 219us) raft_consensus.cc:1520] Early marking committed up to index 2
0807 11:10:35.554363 (+1192364us) log.cc:842] Serialized 14 byte log entry
0807 11:10:35.710112 (+155749us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:35.712500 (+ 2388us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:35.743520 (+ 31020us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:35.743633 (+ 113us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:35.743782 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1001132,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":12715,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":10210,"thread_start_us":1292665,"threads_started":2}
I20260807 11:10:35.753997 25660 catalog_manager.cc:5495] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:10:35.806066 29034 raft_consensus.cc:2781] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:35.806522 29034 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:35.806885 29034 raft_consensus.cc:3037] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:35.807435 28946 consensus_queue.cc:1035] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:10:35.809093 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:35.809518 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 305) took 1016 ms. Trace:
I20260807 11:10:35.810140 25931 raft_consensus.cc:2354] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3.
I20260807 11:10:35.810003 26034 rpcz_store.cc:276] 0807 11:10:34.793198 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:34.798689 (+ 5491us) service_pool.cc:227] Handling call
0807 11:10:34.799024 (+ 335us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:35.585684 (+786660us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:35.585755 (+ 71us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:35.595343 (+ 9588us) log.cc:842] Serialized 26 byte log entry
0807 11:10:35.710245 (+114902us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:35.710325 (+ 80us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:35.809020 (+ 98695us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b54000bbb54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:35.809070 (+ 50us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:35.809140 (+ 70us) raft_consensus.cc:1671] finished
0807 11:10:35.809156 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:35.809425 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":396,"raft.queue_time_us":399361,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":172,"spinlock_wait_cycles":357868928,"thread_start_us":737342,"threads_started":2,"wal-append.queue_time_us":153996}
I20260807 11:10:35.814864 29001 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:35.818827 29034 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:35.820845 25750 leader_election.cc:302] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:35.821152 29034 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:35.821728 25660 catalog_manager.cc:5495] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:35.820693 28946 consensus_queue.cc:1035] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:35.784283 28244 consensus_queue.cc:237] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:35.825327 26033 raft_consensus.cc:2396] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260807 11:10:35.852387 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 51610.1us
I20260807 11:10:35.829066 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:35.828722 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:35.835703 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:35.823310 25750 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:35.871176 26041 raft_consensus.cc:3037] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:35.825953 25869 leader_election.cc:302] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:35.823760 25660 catalog_manager.cc:5495] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: config changed from index 2 to 3, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:35.880492 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:35.883111 26035 raft_consensus.cc:2454] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:35.829300 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:35.886725 26037 raft_consensus.cc:2454] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:35.824016 28911 raft_consensus.cc:2781] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:35.830513 25925 raft_consensus.cc:1226] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260807 11:10:35.888332 25801 raft_consensus.cc:2354] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4.
I20260807 11:10:35.889640 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:35.903517 25919 raft_consensus.cc:2454] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:35.829675 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:35.808104 28882 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:35.894021 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 70953us
I20260807 11:10:35.895509 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:35.895828 25654 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:40714 (request call id 225) took 15019 ms (client timeout 15000 ms). Trace:
I20260807 11:10:35.896435 28747 raft_consensus.cc:482] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:35.898569 25851 heartbeater.cc:644] Failed to heartbeat to 127.23.120.190:37321 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.23.120.190:37321 timed out after 15.000s (SENT)
I20260807 11:10:35.899394 29047 raft_consensus.cc:2781] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:35.880472 28385 raft_consensus.cc:482] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:35.900836 25926 raft_consensus.cc:2454] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
W20260807 11:10:35.901758 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 76101.5us
I20260807 11:10:35.908254 26042 raft_consensus.cc:1226] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:35.907702 25808 raft_consensus.cc:2454] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:35.905988 28894 consensus_queue.cc:1035] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 11:10:35.832635 29013 consensus_queue.cc:1035] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
I20260807 11:10:35.914296 29001 raft_consensus.cc:2781] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:35.921625 29001 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:35.922114 29001 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:35.922202 28882 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:35.923123 25921 raft_consensus.cc:1226] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4. Ops: []
W20260807 11:10:35.909173 25654 rpcz_store.cc:269] 0807 11:10:20.885145 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:20.886135 (+ 990us) service_pool.cc:227] Handling call
0807 11:10:35.895475 (+15009340us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 11:10:35.264554 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 11:10:35.316809 (+ 52255us) write_op.cc:432] Acquiring schema lock in shared mode
0807 11:10:35.317430 (+ 621us) write_op.cc:435] Acquired schema lock
0807 11:10:35.317930 (+ 500us) tablet.cc:660] Decoding operations
0807 11:10:35.352881 (+ 34951us) write_op.cc:620] Acquiring the partition lock for write op
0807 11:10:35.360104 (+ 7223us) write_op.cc:641] Partition lock acquired for write op
0807 11:10:35.360692 (+ 588us) tablet.cc:683] Acquiring locks for 1 operations
0807 11:10:35.365571 (+ 4879us) tablet.cc:699] Row locks acquired
0807 11:10:35.365602 (+ 31us) write_op.cc:260] PREPARE: finished
0807 11:10:35.365809 (+ 207us) write_op.cc:270] Start()
0807 11:10:35.365951 (+ 142us) write_op.cc:276] Timestamp: P: 1786101035365747 usec, L: 0
0807 11:10:35.365978 (+ 27us) op_driver.cc:331] REPLICATION: starting
0807 11:10:35.366865 (+ 887us) log.cc:842] Serialized 539 byte log entry
0807 11:10:35.421692 (+ 54827us) op_driver.cc:444] REPLICATION: finished
0807 11:10:35.422924 (+ 1232us) write_op.cc:301] APPLY: starting
0807 11:10:35.423016 (+ 92us) tablet.cc:1365] starting BulkCheckPresence
0807 11:10:35.423132 (+ 116us) tablet.cc:1368] finished BulkCheckPresence
0807 11:10:35.423271 (+ 139us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 11:10:35.423515 (+ 244us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 11:10:35.423710 (+ 195us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0807 11:10:35.423770 (+ 60us) write_op.cc:312] APPLY: finished
0807 11:10:35.423995 (+ 225us) log.cc:842] Serialized 23 byte log entry
0807 11:10:35.424203 (+ 208us) write_op.cc:489] Releasing partition, row and schema locks
0807 11:10:35.424305 (+ 102us) write_op.cc:454] Released schema lock
0807 11:10:35.424349 (+ 44us) write_op.cc:341] FINISH: Updating metrics
0807 11:10:35.424658 (+ 309us) write_op.cc:489] Releasing partition, row and schema locks
0807 11:10:35.424688 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":338550,"reported_tablets":14,"child_traces":[["op",{"apply.queue_time_us":954,"apply.run_cpu_time_us":2003,"apply.run_wall_time_us":2003,"num_ops":1,"prepare.queue_time_us":1197973,"prepare.run_cpu_time_us":5198,"prepare.run_wall_time_us":156041,"raft.queue_time_us":5975,"raft.run_cpu_time_us":1125,"raft.run_wall_time_us":126189,"replication_time_us":55633,"thread_start_us":485906,"threads_started":4,"wal-append.queue_time_us":46166}]]}
I20260807 11:10:35.914674 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:35.917305 28747 raft_consensus.cc:504] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:35.918346 25869 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:35.918911 26938 leader_election.cc:302] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:35.910823 25916 raft_consensus.cc:2454] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:35.929334 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:35.930882 29034 consensus_queue.cc:1035] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 11:10:35.932224 26938 leader_election.cc:302] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:35.933533 29037 raft_consensus.cc:2781] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:35.933980 26938 leader_election.cc:302] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:35.934384 29037 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:35.934535 29034 raft_consensus.cc:2781] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:35.934985 29037 raft_consensus.cc:3037] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:35.935068 29034 raft_consensus.cc:482] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:35.935802 29034 raft_consensus.cc:3037] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:35.936120 25869 leader_election.cc:302] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:35.917322 28385 raft_consensus.cc:504] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:35.913127 28665 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:35.939626 28385 leader_election.cc:288] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:35.940855 28567 raft_consensus.cc:2781] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:35.941365 28567 raft_consensus.cc:482] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:35.941780 28567 raft_consensus.cc:3037] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:35.942605 25750 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:35.943080 26938 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:35.944586 28647 raft_consensus.cc:2781] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:35.945345 28647 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:35.945739 28647 raft_consensus.cc:3037] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:35.945967 29010 raft_consensus.cc:2781] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:35.946455 29010 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:35.913029 29047 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:35.946822 29010 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:35.947261 29047 raft_consensus.cc:3037] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:35.951974 26046 raft_consensus.cc:1261] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:10:35.953135 28747 leader_election.cc:288] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:35.954523 28882 raft_consensus.cc:2781] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:35.955051 28882 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:35.955250 28438 raft_consensus.cc:2781] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:35.955562 28882 raft_consensus.cc:3037] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:35.955763 28438 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:35.956255 28438 raft_consensus.cc:3037] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:35.957252 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:35.958184 25801 raft_consensus.cc:3037] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:35.955461 25869 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:35.959827 28722 raft_consensus.cc:2781] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:35.960363 28722 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:35.960821 28722 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:35.964493 25806 raft_consensus.cc:1226] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:35.964605 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:35.964519 28438 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:35.965698 25796 raft_consensus.cc:2454] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:35.966956 28438 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:35.967869 28503 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:35.939633 28665 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:35.911537 28911 raft_consensus.cc:686] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:35.971974 25872 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:35.973431 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 103697us
I20260807 11:10:35.973824 28438 raft_consensus.cc:2726] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:35.974529 25872 leader_election.cc:302] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:35.976186 26936 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:35.977032 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 51116.4us
I20260807 11:10:35.978768 26936 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:35.981125 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:35.983230 28423 raft_consensus.cc:3037] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:35.983741 29006 consensus_queue.cc:1035] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:35.985666 29013 raft_consensus.cc:2781] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:35.986387 29013 raft_consensus.cc:686] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:35.989640 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:35.990502 25921 raft_consensus.cc:2454] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:35.995615 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:36.003294 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:36.006071 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:36.007005 25808 raft_consensus.cc:2454] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:10:36.006989 28797 raft_consensus.cc:2781] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:36.009577 28797 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:36.009697 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:36.010094 28797 raft_consensus.cc:3037] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:36.014696 25798 raft_consensus.cc:1226] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:36.015563 25660 catalog_manager.cc:5495] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: term changed from 1 to 3, leader changed from 606d49e053a346e486f48cb31832b815 (127.23.120.129) to eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132). New cstate: current_term: 3 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:10:36.016254 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:35.929872 28449 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:36.021908 28449 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:36.022964 29013 consensus_queue.cc:237] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:36.025969 28882 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:36.026539 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:36.027274 26033 raft_consensus.cc:3037] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:36.027309 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:36.028246 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:36.029975 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:36.031466 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:36.033020 25752 leader_election.cc:302] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:36.034888 28946 raft_consensus.cc:2781] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:36.035225 25869 leader_election.cc:302] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:36.035974 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:36.036164 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:36.036736 25916 raft_consensus.cc:2454] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:36.036963 25796 raft_consensus.cc:2454] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:36.035346 28946 raft_consensus.cc:482] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:36.037652 28946 raft_consensus.cc:3037] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:36.038143 25926 raft_consensus.cc:1226] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260807 11:10:36.038756 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:36.039533 26037 raft_consensus.cc:2454] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:36.039752 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:36.040832 25918 raft_consensus.cc:2454] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:36.041914 25917 raft_consensus.cc:1226] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260807 11:10:36.043154 28665 raft_consensus.cc:2781] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:36.043808 28665 raft_consensus.cc:686] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 3 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:36.045107 28665 consensus_queue.cc:237] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:36.045815 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:36.047541 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:36.036371 25869 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:36.050330 29052 raft_consensus.cc:2741] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260807 11:10:36.054682 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:36.007375 29006 consensus_queue.cc:1046] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20260807 11:10:36.057152 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:36.057099 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:36.058209 25926 raft_consensus.cc:2379] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 3: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:36.058635 25796 raft_consensus.cc:2454] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:36.060860 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:35.973433 28911 consensus_queue.cc:237] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:36.065155 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:36.065860 29006 raft_consensus.cc:3032] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:36.065958 25798 raft_consensus.cc:2454] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:36.066461 29006 raft_consensus.cc:729] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:36.064431 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:36.064435 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:36.068470 25919 raft_consensus.cc:2454] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
W20260807 11:10:36.078644 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 132711us
I20260807 11:10:36.082908 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:36.085305 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:36.101348 26033 raft_consensus.cc:2454] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:36.099367 25926 raft_consensus.cc:2454] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:36.102921 25801 raft_consensus.cc:2454] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:36.104719 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:36.104751 25918 raft_consensus.cc:1226] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20260807 11:10:36.109383 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:36.111287 25921 raft_consensus.cc:1226] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260807 11:10:36.113070 25660 catalog_manager.cc:5495] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: term changed from 2 to 3, leader changed from 606d49e053a346e486f48cb31832b815 (127.23.120.129) to eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132). New cstate: current_term: 3 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:10:36.115687 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:36.117105 25981 leader_election.cc:334] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195): Timed out: RequestConsensusVote RPC to 127.23.120.129:40195 timed out after 1.562s (SENT)
I20260807 11:10:36.126416 25981 leader_election.cc:302] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:36.126930 25916 raft_consensus.cc:2379] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 3: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
W20260807 11:10:36.127652 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 153245us
I20260807 11:10:36.108429 28503 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:36.136718 25917 raft_consensus.cc:2454] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:36.151782 25750 leader_election.cc:302] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:36.154829 25750 leader_election.cc:302] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:36.155876 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:36.156600 25801 raft_consensus.cc:2454] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 5.
I20260807 11:10:36.157452 25932 raft_consensus.cc:1226] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260807 11:10:36.157574 25869 leader_election.cc:302] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:36.097868 28722 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:36.172725 25752 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:36.179929 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:36.176972 25869 leader_election.cc:302] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:36.171527 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:36.182579 25916 raft_consensus.cc:3037] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:36.183388 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:36.138831 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:36.184410 25800 raft_consensus.cc:1226] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:36.184682 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:36.190265 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 73218.7us
I20260807 11:10:36.164853 28438 raft_consensus.cc:2781] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:36.197935 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:36.199222 25921 raft_consensus.cc:1226] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:36.167946 28244 raft_consensus.cc:2781] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:36.168359 28747 raft_consensus.cc:3037] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:36.201683 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:36.206670 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:36.208837 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:36.200101 25808 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:36.216391 25799 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:36.217170 25917 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:36.198374 28438 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:36.219398 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:36.164239 29010 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:36.163103 28567 raft_consensus.cc:504] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:36.222398 26037 raft_consensus.cc:2379] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 3: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:36.226462 26039 raft_consensus.cc:2454] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:36.183861 29052 consensus_queue.cc:1035] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260807 11:10:36.204510 28244 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:36.179102 29006 consensus_queue.cc:260] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:36.223816 28293 raft_consensus.cc:2781] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:36.231439 25797 raft_consensus.cc:3037] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
W20260807 11:10:36.224206 29077 log.cc:924] Time spent T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.102s user 0.000s sys 0.001s
I20260807 11:10:36.224797 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:36.233054 28882 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:36.235277 28385 raft_consensus.cc:482] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:36.235844 28385 raft_consensus.cc:504] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:36.237915 28385 leader_election.cc:288] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:36.239678 28567 leader_election.cc:288] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:36.239316 29001 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:36.227880 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:36.241905 25919 raft_consensus.cc:2379] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 5: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:36.212823 28911 raft_consensus.cc:2726] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 11:10:36.214361 28462 raft_consensus.cc:2726] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 11:10:36.210237 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:36.229334 28244 raft_consensus.cc:3037] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:36.242475 26036 raft_consensus.cc:2379] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 4: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:36.241075 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:36.225782 28438 raft_consensus.cc:3037] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
W20260807 11:10:36.220903 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 59671.7us
I20260807 11:10:36.234201 28293 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:36.217531 29037 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:36.244109 25660 catalog_manager.cc:5495] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: term changed from 1 to 3, leader changed from 606d49e053a346e486f48cb31832b815 (127.23.120.129) to eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132). New cstate: current_term: 3 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:10:36.217646 29047 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:36.234010 25981 connection.cc:232] RPC call timeout handler was delayed by -2.5600613679271191s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25510000000000005s for a response
I20260807 11:10:36.204079 28761 consensus_queue.cc:1046] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260807 11:10:36.263891 28293 raft_consensus.cc:3037] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
W20260807 11:10:36.269824 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 104067us
W20260807 11:10:36.283034 29089 log.cc:924] Time spent T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260807 11:10:36.309679 29001 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:36.310415 29084 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:36.311311 25869 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:36.287055 29010 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:36.309784 28946 raft_consensus.cc:504] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:36.313282 29092 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:36.313974 29092 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:36.314571 28946 leader_election.cc:288] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:36.261426 28647 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:36.315963 25797 raft_consensus.cc:2454] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
W20260807 11:10:36.283596 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 128871us
I20260807 11:10:36.317512 29001 raft_consensus.cc:2726] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:36.317837 25750 leader_election.cc:302] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:36.265070 25872 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:36.319128 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 88187.2us
I20260807 11:10:36.311082 29084 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:36.317584 28722 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:36.321013 29006 raft_consensus.cc:3037] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:36.321691 29084 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:36.321859 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:36.322588 26036 raft_consensus.cc:3037] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:36.322952 25872 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:36.323849 28722 raft_consensus.cc:2726] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:36.304519 29037 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:36.325415 25872 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:36.326334 28722 raft_consensus.cc:2781] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:36.326759 28722 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:36.327106 28722 raft_consensus.cc:3037] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:36.319739 29001 raft_consensus.cc:2726] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:36.254930 28797 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:36.233444 29034 raft_consensus.cc:504] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:36.317549 28647 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:36.290836 28293 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:36.292886 25916 raft_consensus.cc:2454] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:10:36.333235 26041 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:36.349092 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59312 (request call id 1558) took 1141 ms. Trace:
I20260807 11:10:36.329715 25660 catalog_manager.cc:5495] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: term changed from 1 to 3, leader changed from b22d69db45194908ad3d345eba9badd9 (127.23.120.131) to eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132). New cstate: current_term: 3 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:10:36.336653 25799 raft_consensus.cc:2454] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:10:36.267256 28423 raft_consensus.cc:2726] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:36.349586 26041 rpcz_store.cc:276] 0807 11:10:35.207884 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:35.776716 (+568832us) service_pool.cc:227] Handling call
0807 11:10:36.349015 (+572299us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:36.355196 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:10:36.360072 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 78080.3us
I20260807 11:10:36.364964 25917 raft_consensus.cc:2454] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
W20260807 11:10:36.369125 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 57907.2us
I20260807 11:10:36.373335 25660 catalog_manager.cc:5495] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: term changed from 1 to 3, leader changed from b22d69db45194908ad3d345eba9badd9 (127.23.120.131) to eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132). New cstate: current_term: 3 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:10:36.381774 25981 leader_election.cc:302] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:36.367386 29092 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:36.381744 25660 catalog_manager.cc:5495] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: term changed from 2 to 3, leader changed from b22d69db45194908ad3d345eba9badd9 (127.23.120.131) to eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132). New cstate: current_term: 3 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:10:36.389822 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:36.390924 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:36.391961 25872 leader_election.cc:302] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:36.393153 25752 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:10:36.393290 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 75523.5us
I20260807 11:10:36.396806 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:36.397354 25750 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:36.399694 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:36.401711 26037 raft_consensus.cc:3037] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:36.402175 25869 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:36.381359 28911 raft_consensus.cc:2726] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:36.381103 28647 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:36.405205 25660 catalog_manager.cc:5495] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 reported cstate change: term changed from 1 to 2, leader changed from 606d49e053a346e486f48cb31832b815 (127.23.120.129) to eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132). New cstate: current_term: 2 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:10:36.413026 26039 raft_consensus.cc:3037] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:36.395040 29029 raft_consensus.cc:2741] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260807 11:10:36.404433 28964 raft_consensus.cc:2726] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:36.439328 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:36.441314 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 44894.4us
I20260807 11:10:36.440047 26034 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
W20260807 11:10:36.435467 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 57177.4us
I20260807 11:10:36.447814 25869 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:36.442333 28665 raft_consensus.cc:979] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc: : Instructing follower b22d69db45194908ad3d345eba9badd9 to start an election
I20260807 11:10:36.450615 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:36.434758 28911 raft_consensus.cc:2781] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election won for term 4
W20260807 11:10:36.453537 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 79777.6us
I20260807 11:10:36.445379 28882 raft_consensus.cc:2781] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:36.465560 26986 raft_consensus.cc:2354] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4.
I20260807 11:10:36.475361 26037 raft_consensus.cc:2454] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:10:36.449745 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:36.450466 28462 raft_consensus.cc:2726] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:36.475742 25981 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:36.481266 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:36.485165 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:36.485405 25750 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:36.446694 28647 raft_consensus.cc:504] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:36.486593 26938 leader_election.cc:302] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:36.490422 28911 raft_consensus.cc:686] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:36.479717 29034 leader_election.cc:288] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:36.476549 28747 raft_consensus.cc:2726] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:36.474270 28797 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:36.484472 28293 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:36.491070 26984 raft_consensus.cc:2379] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 3: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:36.490594 26979 raft_consensus.cc:2379] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 4: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:36.489691 28665 raft_consensus.cc:2726] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:36.488020 29118 consensus_queue.cc:1046] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
W20260807 11:10:36.513571 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 70314.2us
W20260807 11:10:36.573501 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 87041.1us
W20260807 11:10:36.574024 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 144636us
I20260807 11:10:36.480227 29052 raft_consensus.cc:1067] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Signalling peer b22d69db45194908ad3d345eba9badd9 to start an election
I20260807 11:10:36.480886 28882 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:36.577221 26036 raft_consensus.cc:2454] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:10:36.582201 26038 raft_consensus.cc:2454] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
W20260807 11:10:36.577314 29102 log.cc:924] Time spent T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.136s user 0.000s sys 0.001s
W20260807 11:10:36.584197 29107 log.cc:924] Time spent T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.090s user 0.000s sys 0.008s
W20260807 11:10:36.685350 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 225879us
I20260807 11:10:36.710917 29096 raft_consensus.cc:3032] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Stepping down as leader of term 3
W20260807 11:10:36.711084 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 222416us
I20260807 11:10:36.711452 29096 raft_consensus.cc:729] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:36.713294 25981 leader_election.cc:398] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195) with higher term. Message: Invalid argument: T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4.
I20260807 11:10:36.714583 25981 leader_election.cc:401] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260807 11:10:36.663756 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:36.719024 26938 leader_election.cc:302] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:36.720458 25869 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:36.567655 29047 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:36.709686 29114 log.cc:924] Time spent T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.110s user 0.000s sys 0.000s
W20260807 11:10:36.670684 29106 log.cc:924] Time spent T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.093s user 0.000s sys 0.000s
W20260807 11:10:36.673077 29108 log.cc:924] Time spent T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.089s user 0.000s sys 0.008s
I20260807 11:10:36.724499 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:36.725344 28647 leader_election.cc:288] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:36.721275 29037 raft_consensus.cc:2781] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:36.742471 28722 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:36.743296 29037 raft_consensus.cc:686] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:36.666440 28293 raft_consensus.cc:2726] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 11:10:36.744318 29037 consensus_queue.cc:237] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:36.707955 29117 consensus_queue.cc:1035] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.143s
I20260807 11:10:36.725669 28911 consensus_queue.cc:237] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:36.717085 28882 raft_consensus.cc:3037] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
W20260807 11:10:36.726974 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 261546us
I20260807 11:10:36.744720 28722 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:36.602586 28747 raft_consensus.cc:2781] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:36.731832 26039 raft_consensus.cc:2454] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:10:36.752352 25750 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:36.760119 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:36.758635 25869 leader_election.cc:302] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:36.761368 26975 raft_consensus.cc:2379] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 3: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:36.762004 28964 raft_consensus.cc:2726] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:36.749460 29029 raft_consensus.cc:3032] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:36.762750 29029 raft_consensus.cc:729] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:36.763679 29029 consensus_queue.cc:260] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:36.765254 25750 leader_election.cc:302] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:36.765518 29029 raft_consensus.cc:3037] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:36.766284 28946 raft_consensus.cc:2726] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:36.767306 25798 raft_consensus.cc:2454] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:36.768450 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:36.771211 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:36.771976 25926 raft_consensus.cc:2379] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 3: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:36.755084 28293 consensus_queue.cc:1035] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20260807 11:10:36.728108 25981 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:36.728641 29118 raft_consensus.cc:2726] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195) with higher term. Message: Invalid argument: T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4.
I20260807 11:10:36.728382 28244 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:36.728797 28385 consensus_queue.cc:1046] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
W20260807 11:10:36.783044 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 55026.7us
I20260807 11:10:36.731889 25872 leader_election.cc:302] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:36.733006 29096 consensus_queue.cc:260] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:36.723600 28449 raft_consensus.cc:2781] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:36.788676 28449 raft_consensus.cc:686] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:36.789712 28449 consensus_queue.cc:237] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:36.792132 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:36.733831 28438 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:36.792896 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:36.793903 26985 raft_consensus.cc:1226] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260807 11:10:36.794414 28438 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:36.795248 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:36.792918 26986 raft_consensus.cc:2354] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4.
I20260807 11:10:36.796792 28385 raft_consensus.cc:3032] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:36.797168 28385 raft_consensus.cc:729] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:36.757400 25752 leader_election.cc:302] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:36.730219 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:36.786659 26936 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:36.762180 28722 raft_consensus.cc:2781] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:36.785722 29117 raft_consensus.cc:2781] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:36.824536 29117 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:36.824972 29117 raft_consensus.cc:3037] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:36.801981 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:36.803807 29096 raft_consensus.cc:3037] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:36.828217 26979 raft_consensus.cc:2379] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 3: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:36.805910 28244 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:36.830773 28244 raft_consensus.cc:2781] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:36.831218 28244 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:36.831699 28244 raft_consensus.cc:3037] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:36.809083 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:36.777271 28665 raft_consensus.cc:2726] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:36.840903 29001 consensus_queue.cc:1046] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 11:10:36.841477 25918 raft_consensus.cc:2379] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 6: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
W20260807 11:10:36.811910 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 80161us
I20260807 11:10:36.843091 25872 leader_election.cc:302] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:36.844091 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:36.844455 28797 raft_consensus.cc:3032] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 5 LEADER]: Stepping down as leader of term 5
I20260807 11:10:36.844874 28797 raft_consensus.cc:729] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:36.845710 28797 consensus_queue.cc:260] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:36.846969 29099 raft_consensus.cc:2726] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:36.812403 29052 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:36.849936 29052 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:36.850502 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:36.850658 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:36.851537 25925 raft_consensus.cc:2379] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 4: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:36.852128 29052 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:36.852460 26979 raft_consensus.cc:1226] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:36.812999 28438 raft_consensus.cc:2781] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:36.853072 28438 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:36.853477 28438 raft_consensus.cc:3037] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:36.813544 28722 raft_consensus.cc:686] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 3 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:36.848846 25872 leader_election.cc:302] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:36.858078 26936 leader_election.cc:302] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:36.759282 28747 raft_consensus.cc:686] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
W20260807 11:10:36.815353 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 58027.7us
I20260807 11:10:36.817562 25981 leader_election.cc:302] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:36.819119 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:36.859769 25926 raft_consensus.cc:2454] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:36.819979 28567 raft_consensus.cc:2726] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:36.820874 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:36.868575 25869 leader_election.cc:398] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4.
I20260807 11:10:36.868932 25869 leader_election.cc:401] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260807 11:10:36.821040 26938 leader_election.cc:302] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:36.797995 28385 consensus_queue.cc:260] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:36.876077 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 58600.5us
I20260807 11:10:36.876398 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:36.837285 26975 raft_consensus.cc:2379] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 6: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:36.883186 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:36.873200 26936 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:36.874720 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:36.884583 26986 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
W20260807 11:10:36.881992 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 61027.5us
I20260807 11:10:36.885576 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:36.886749 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:36.887838 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:36.888600 25799 raft_consensus.cc:3037] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 4
I20260807 11:10:36.889501 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:36.890141 25808 raft_consensus.cc:3037] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:36.898841 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:36.899655 25808 raft_consensus.cc:2454] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:36.854996 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:36.898005 25752 leader_election.cc:302] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:10:36.901213 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 52468.1us
I20260807 11:10:36.904860 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:36.905628 26984 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:36.868126 29052 raft_consensus.cc:2726] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:36.912443 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:36.913336 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:36.914248 25925 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:36.920318 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:36.897925 29034 raft_consensus.cc:2781] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:36.901379 28567 raft_consensus.cc:2781] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:36.928412 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:36.930609 25981 leader_election.cc:302] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:36.897279 28665 raft_consensus.cc:2726] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4.
I20260807 11:10:36.935254 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:36.936045 25750 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:36.943289 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:36.944183 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:36.940337 26984 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:36.959399 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:36.974406 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:36.978528 25872 leader_election.cc:302] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:36.937553 28293 raft_consensus.cc:2781] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:36.942510 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:37.026957 25869 leader_election.cc:302] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:36.998015 25981 leader_election.cc:302] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:36.978590 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:37.103029 28911 raft_consensus.cc:3037] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:37.121215 29029 raft_consensus.cc:2781] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:37.119856 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:37.121758 29029 raft_consensus.cc:482] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
W20260807 11:10:37.122160 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 124258us
I20260807 11:10:37.122450 29029 raft_consensus.cc:3037] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:37.102950 26976 raft_consensus.cc:1226] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:37.123121 25981 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:37.121932 25931 raft_consensus.cc:2379] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 4: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:37.033080 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:37.032621 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:37.033298 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:37.025269 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
W20260807 11:10:37.000937 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 63563.8us
I20260807 11:10:37.006255 25799 raft_consensus.cc:2454] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:37.027398 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
W20260807 11:10:37.105016 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 169223us
I20260807 11:10:37.131788 25800 raft_consensus.cc:2354] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4.
I20260807 11:10:36.937448 29084 raft_consensus.cc:2781] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:10:37.133796 25750 leader_election.cc:302] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:36.983650 25796 raft_consensus.cc:2362] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4. Re-sending same reply.
I20260807 11:10:37.125136 29117 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:37.138576 29129 consensus_queue.cc:1046] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 11:10:37.073403 29122 raft_consensus.cc:2781] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:37.130596 28293 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:37.153693 25921 raft_consensus.cc:1226] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260807 11:10:37.153759 28293 raft_consensus.cc:3037] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:37.055369 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:37.154783 28761 raft_consensus.cc:3032] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:37.155177 28761 raft_consensus.cc:729] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:37.155966 28761 consensus_queue.cc:260] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:37.157238 28761 raft_consensus.cc:3037] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:37.157188 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:36.960059 28882 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:37.159193 28244 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:36.999078 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:37.027943 28722 consensus_queue.cc:237] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:37.131418 25803 raft_consensus.cc:2454] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
W20260807 11:10:37.051985 29108 log.cc:924] Time spent T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20260807 11:10:37.143313 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
W20260807 11:10:37.147095 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 204386us
I20260807 11:10:37.149178 25786 raft_consensus.cc:1226] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:37.149641 25750 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:37.150259 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:37.149992 29100 consensus_queue.cc:1046] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 11:10:37.150782 25808 raft_consensus.cc:1261] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:10:36.972924 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:36.977269 28567 raft_consensus.cc:686] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:37.151222 29129 raft_consensus.cc:2726] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:37.027297 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:37.152081 29052 raft_consensus.cc:2781] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:37.152259 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:37.149832 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:37.140645 28665 raft_consensus.cc:2781] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:37.184095 28665 raft_consensus.cc:482] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:37.184556 28665 raft_consensus.cc:3037] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:37.185320 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:37.186084 25921 raft_consensus.cc:2379] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 4: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:37.188102 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:37.153214 29122 raft_consensus.cc:482] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:37.188864 25920 raft_consensus.cc:2379] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 4: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:37.188969 29122 raft_consensus.cc:3037] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:37.189268 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:37.189957 25932 raft_consensus.cc:3037] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:37.190043 25920 raft_consensus.cc:1226] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260807 11:10:37.124995 28797 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:37.196905 29006 raft_consensus.cc:2781] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:37.197139 29052 raft_consensus.cc:686] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:37.197521 29006 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:37.198104 29006 raft_consensus.cc:3037] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:37.198295 29052 consensus_queue.cc:237] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:37.199359 25801 raft_consensus.cc:1261] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 11:10:37.201520 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:37.204027 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:37.204909 25797 raft_consensus.cc:2379] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 4: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:37.206588 28882 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:37.211037 29141 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:37.211684 29141 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:37.062165 28747 consensus_queue.cc:237] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:37.214031 29141 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:37.214435 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:37.215234 25750 leader_election.cc:302] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:37.179762 28244 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:37.151803 29140 raft_consensus.cc:2781] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:37.216156 29140 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:37.216568 29140 raft_consensus.cc:3037] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:37.218878 29117 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:37.220498 26037 raft_consensus.cc:1261] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 11:10:37.178093 29084 raft_consensus.cc:686] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:37.222522 29084 consensus_queue.cc:237] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:37.198798 29029 raft_consensus.cc:504] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:37.182790 25981 leader_election.cc:302] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:37.212020 28894 raft_consensus.cc:3032] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:37.241181 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:37.241637 28894 raft_consensus.cc:729] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:37.178604 25796 raft_consensus.cc:3037] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:37.131721 28385 raft_consensus.cc:3037] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:37.133491 29034 raft_consensus.cc:686] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
W20260807 11:10:37.147779 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 120930us
W20260807 11:10:37.229681 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 47035.3us
I20260807 11:10:37.226222 25798 raft_consensus.cc:2454] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:37.254004 29092 consensus_queue.cc:1035] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 11:10:37.254824 25797 raft_consensus.cc:1261] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 11:10:37.256445 25801 raft_consensus.cc:1261] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 11:10:37.245200 25872 leader_election.cc:302] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:37.256386 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:37.251243 25932 raft_consensus.cc:2454] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:37.271346 25932 raft_consensus.cc:1226] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260807 11:10:37.281198 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:37.288985 26033 raft_consensus.cc:1226] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:37.289153 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:37.293550 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:37.293843 25750 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:37.294397 26046 raft_consensus.cc:3037] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
W20260807 11:10:37.289188 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 140304us
I20260807 11:10:37.141566 26978 raft_consensus.cc:2454] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:37.307092 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:37.312458 25920 raft_consensus.cc:2396] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260807 11:10:37.313184 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 178933us
I20260807 11:10:37.268828 28946 raft_consensus.cc:2726] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:37.318636 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:37.280793 28882 raft_consensus.cc:482] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:37.318015 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:37.315514 25981 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:37.319690 25932 raft_consensus.cc:2379] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 4: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:37.211014 29006 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:37.242591 28894 consensus_queue.cc:260] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:37.252229 29122 raft_consensus.cc:504] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:37.280858 28438 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:37.289664 28665 raft_consensus.cc:504] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:37.351868 25932 raft_consensus.cc:1226] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:37.295980 29118 raft_consensus.cc:2741] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260807 11:10:37.319320 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:37.322880 25921 raft_consensus.cc:1261] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 11:10:37.388291 28747 consensus_queue.cc:1046] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.012s
I20260807 11:10:37.320041 26936 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:37.329838 25919 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
W20260807 11:10:37.405612 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 160521us
I20260807 11:10:37.406834 25872 leader_election.cc:302] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:37.408270 25918 raft_consensus.cc:2379] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 3: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:37.314772 29047 raft_consensus.cc:2726] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260807 11:10:37.410000 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 90081.2us
I20260807 11:10:37.314253 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:37.357215 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:37.357790 25750 leader_election.cc:302] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:37.421736 29143 consensus_queue.cc:1046] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 11:10:37.392184 28797 raft_consensus.cc:2726] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:37.425173 26936 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:37.436628 28747 consensus_queue.cc:1225] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer 606d49e053a346e486f48cb31832b815 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260807 11:10:37.436090 25981 leader_election.cc:302] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:37.329000 29117 raft_consensus.cc:2781] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:37.413311 26976 raft_consensus.cc:3037] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:37.408133 28964 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
W20260807 11:10:37.453576 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 95861us
I20260807 11:10:37.339529 28882 raft_consensus.cc:504] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:37.472491 25750 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:37.474419 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:10:37.438021 28647 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:37.481904 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 70693.1us
I20260807 11:10:37.481634 26046 raft_consensus.cc:2454] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
W20260807 11:10:37.482868 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 57797.6us
I20260807 11:10:37.487506 25752 leader_election.cc:302] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:37.501855 29098 consensus_queue.cc:1046] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.151s
W20260807 11:10:37.503515 29139 log.cc:924] Time spent T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.174s user 0.000s sys 0.001s
W20260807 11:10:37.504755 29156 log.cc:924] Time spent T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.090s user 0.001s sys 0.000s
I20260807 11:10:37.507006 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:37.385955 29034 consensus_queue.cc:237] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:37.452696 29117 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:37.508209 25926 raft_consensus.cc:2454] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:37.510773 25750 leader_election.cc:302] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:37.350811 28567 consensus_queue.cc:237] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:37.485809 29047 raft_consensus.cc:2726] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:37.485512 28747 raft_consensus.cc:2741] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260807 11:10:37.485144 29029 leader_election.cc:288] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:37.498601 25799 raft_consensus.cc:2454] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:10:37.512226 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:37.566538 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 59784.1us
I20260807 11:10:37.566850 25916 raft_consensus.cc:2396] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260807 11:10:37.514238 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:37.514532 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:37.483975 29037 raft_consensus.cc:2726] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 11:10:37.516597 26936 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:10:37.581269 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 174549us
I20260807 11:10:37.568482 25921 raft_consensus.cc:3037] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:37.444424 29140 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:37.596275 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 96010.9us
I20260807 11:10:37.597954 25750 leader_election.cc:302] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:37.515723 29096 raft_consensus.cc:2781] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:37.519879 29117 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:37.519730 29129 raft_consensus.cc:2781] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:37.602097 25869 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:37.609938 25931 raft_consensus.cc:1226] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:37.610188 25920 raft_consensus.cc:2454] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:37.485301 28964 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:37.619408 26033 raft_consensus.cc:1226] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:37.509812 28647 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:37.622865 25750 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:37.600841 28438 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:37.601547 28761 raft_consensus.cc:2726] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:37.601842 29096 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:37.620863 29052 raft_consensus.cc:2781] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:37.605566 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:37.632936 28882 leader_election.cc:288] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:37.633213 29144 log.cc:924] Time spent T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.197s user 0.000s sys 0.001s
I20260807 11:10:37.633126 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:37.633467 29176 consensus_queue.cc:1046] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20260807 11:10:37.633782 29127 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:37.634251 29119 raft_consensus.cc:2726] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260807 11:10:37.634493 29166 log.cc:924] Time spent T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20260807 11:10:37.634637 29170 raft_consensus.cc:2726] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:37.636368 28964 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:37.637460 29191 consensus_queue.cc:1046] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:37.638167 28946 raft_consensus.cc:2781] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:37.638960 29122 leader_election.cc:288] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:37.639333 26039 raft_consensus.cc:3032] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:37.641023 29096 raft_consensus.cc:3037] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:37.715668 26039 raft_consensus.cc:729] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:37.716972 29118 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:37.717147 28964 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:37.717556 29118 raft_consensus.cc:504] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:37.632568 25750 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:37.618655 29129 raft_consensus.cc:686] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:37.643512 25872 leader_election.cc:302] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:37.720309 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 76872.4us
I20260807 11:10:37.720229 29129 consensus_queue.cc:237] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:37.721238 25872 leader_election.cc:302] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:37.722378 25872 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:37.723119 25872 leader_election.cc:302] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:37.604199 29037 consensus_queue.cc:1046] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20260807 11:10:37.626170 28665 leader_election.cc:288] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:37.626683 29029 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:37.688012 25981 leader_election.cc:398] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4.
I20260807 11:10:37.630066 29141 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:37.693753 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:37.649519 29052 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
W20260807 11:10:37.698976 29147 log.cc:924] Time spent T 00000000000000000000000000000000 P 98664bdefe2a4a829847688b8d0d50c1: Append to log took a long time: real 0.153s user 0.000s sys 0.004s
I20260807 11:10:37.704664 29176 consensus_queue.cc:1225] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer eb1fca62dcb34ed9b32188ccb19f0543 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260807 11:10:37.741778 26042 raft_consensus.cc:1261] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260807 11:10:37.701794 28293 consensus_meta.cc:287] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.545s user 0.001s sys 0.009s
I20260807 11:10:37.712575 28946 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:37.717658 28964 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:37.618539 29143 raft_consensus.cc:2781] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:37.727643 28438 raft_consensus.cc:2726] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260807 11:10:37.682583 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 50880.9us
W20260807 11:10:37.731473 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 99017us
I20260807 11:10:37.716612 26039 consensus_queue.cc:260] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:37.700825 29127 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:37.733333 29170 raft_consensus.cc:2726] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 11:10:37.733462 29052 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:37.783767 25872 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:37.745021 29141 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:37.713275 29006 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:37.683244 28747 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:37.791127 26039 raft_consensus.cc:3037] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:37.747936 25796 raft_consensus.cc:2454] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:10:37.749141 25981 leader_election.cc:302] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:37.755541 25750 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:37.756021 29193 raft_consensus.cc:3032] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:37.734249 29047 raft_consensus.cc:2726] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 11:10:37.788005 28385 consensus_meta.cc:287] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.515s user 0.010s sys 0.002s
I20260807 11:10:37.752691 29143 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:37.749082 28293 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:37.751204 28946 raft_consensus.cc:3037] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:37.732265 29029 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:37.757860 29172 log.cc:924] Time spent T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.074s user 0.000s sys 0.001s
W20260807 11:10:37.759083 28911 consensus_meta.cc:287] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.638s user 0.008s sys 0.003s
I20260807 11:10:37.760941 29140 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:37.768294 29184 raft_consensus.cc:3032] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:37.768586 29117 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:37.772843 29142 raft_consensus.cc:3032] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:37.733114 28665 raft_consensus.cc:2781] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:37.790071 25869 leader_election.cc:398] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Vote denied by peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195) with higher term. Message: Invalid argument: T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4.
I20260807 11:10:37.795938 25750 leader_election.cc:302] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:37.793869 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 277332us
W20260807 11:10:37.798678 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 49600.1us
W20260807 11:10:37.860679 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 64820.3us
I20260807 11:10:37.790577 25921 raft_consensus.cc:2454] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:37.756932 28964 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:37.795099 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:37.807296 25799 raft_consensus.cc:1261] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 11:10:37.809295 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:37.823150 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:37.794821 29193 raft_consensus.cc:729] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:37.824177 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:37.845256 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:37.851099 26039 raft_consensus.cc:1261] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 11:10:37.843021 25806 raft_consensus.cc:1226] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:37.874104 25786 raft_consensus.cc:3037] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:37.810407 28722 raft_consensus.cc:2781] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:37.811542 29143 raft_consensus.cc:3037] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:37.814651 29191 raft_consensus.cc:3032] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:37.856530 26046 raft_consensus.cc:3037] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:37.817881 28244 raft_consensus.cc:2726] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 11:10:37.862298 25750 leader_election.cc:302] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:37.863766 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 73907.6us
I20260807 11:10:37.804265 29084 consensus_queue.cc:1035] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260807 11:10:37.876677 25921 raft_consensus.cc:1226] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:37.878067 25917 raft_consensus.cc:1226] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260807 11:10:37.880154 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 131175us
I20260807 11:10:37.842471 28911 raft_consensus.cc:2726] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:37.880964 25799 raft_consensus.cc:1226] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:37.887604 26034 raft_consensus.cc:1226] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 5. Current term is 6. Ops: []
I20260807 11:10:37.890092 25750 leader_election.cc:302] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:37.896363 29181 raft_consensus.cc:2726] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:37.897039 29169 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:37.890512 29129 raft_consensus.cc:2726] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:37.895820 29094 consensus_queue.cc:1046] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.017s
I20260807 11:10:37.901335 29094 raft_consensus.cc:3032] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:37.901770 29094 raft_consensus.cc:729] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:37.902899 29094 consensus_queue.cc:260] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:37.896903 29119 raft_consensus.cc:2726] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 11:10:37.904609 29094 raft_consensus.cc:3037] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:37.909719 25750 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:37.855140 28647 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:37.911028 25750 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:37.913120 29119 raft_consensus.cc:2726] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 11:10:37.917757 29141 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:37.917369 29193 consensus_queue.cc:260] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:37.887159 29191 raft_consensus.cc:729] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:37.918803 29193 raft_consensus.cc:3037] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:37.919071 29191 consensus_queue.cc:260] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:37.919879 28894 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:37.920421 29191 raft_consensus.cc:3037] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
W20260807 11:10:37.922063 29123 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260807 11:10:37.923919 29118 leader_election.cc:288] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:37.925181 29127 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:37.927753 25797 raft_consensus.cc:1226] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260807 11:10:37.929111 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 177798us
I20260807 11:10:37.897660 29169 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:37.898272 28293 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:37.862085 29173 raft_consensus.cc:2741] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260807 11:10:37.854233 29184 raft_consensus.cc:729] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:37.899708 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:37.936869 26038 raft_consensus.cc:3037] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:37.886896 25869 leader_election.cc:302] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:37.895455 25981 leader_election.cc:398] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 election: Vote denied by peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465) with higher term. Message: Invalid argument: T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3.
I20260807 11:10:37.938651 25981 leader_election.cc:401] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260807 11:10:37.934885 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:37.935825 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:37.858655 29142 raft_consensus.cc:729] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:37.804004 28747 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:37.938033 25803 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
W20260807 11:10:37.936743 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 47526.4us
I20260807 11:10:37.943048 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:37.933059 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:37.928557 26046 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:37.948325 25921 raft_consensus.cc:1226] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:37.950028 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:37.883723 28964 raft_consensus.cc:2726] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 11:10:37.950464 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:37.993767 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:37.993007 26936 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:37.879776 28665 raft_consensus.cc:686] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:37.992138 26984 raft_consensus.cc:1226] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:38.001019 25919 raft_consensus.cc:1261] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 11:10:38.019388 25797 raft_consensus.cc:1261] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 11:10:38.001192 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:38.021862 26042 raft_consensus.cc:3037] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:37.994042 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:38.001922 25917 raft_consensus.cc:1226] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260807 11:10:37.993744 26041 raft_consensus.cc:1261] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 11:10:38.004268 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:38.005268 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:38.024947 25926 raft_consensus.cc:2454] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:38.005604 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:37.936587 28722 raft_consensus.cc:686] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 3 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:38.035554 25786 raft_consensus.cc:2454] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:38.036345 28722 consensus_queue.cc:237] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:38.041123 25798 raft_consensus.cc:1226] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:38.007014 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:10:38.007345 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 120548us
I20260807 11:10:38.008355 26985 raft_consensus.cc:2379] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 6: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:37.936924 28647 raft_consensus.cc:2781] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:38.048061 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:38.009155 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:38.005292 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:38.011724 25920 raft_consensus.cc:2379] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 4: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:38.007165 28438 raft_consensus.cc:2781] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:37.869894 28567 consensus_queue.cc:1035] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.120s
I20260807 11:10:38.014839 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:38.055154 25916 raft_consensus.cc:2454] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:38.019289 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:38.019371 26936 leader_election.cc:302] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:38.025216 25932 raft_consensus.cc:2379] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 3: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:38.006212 29118 raft_consensus.cc:2726] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465) with higher term. Message: Invalid argument: T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3.
I20260807 11:10:37.929685 29006 consensus_queue.cc:1035] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.078s
I20260807 11:10:38.031052 28665 consensus_queue.cc:237] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:38.018839 29194 raft_consensus.cc:2726] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:38.001315 29013 consensus_queue.cc:1035] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260807 11:10:38.049616 26035 raft_consensus.cc:2362] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3. Re-sending same reply.
I20260807 11:10:38.006430 28423 consensus_queue.cc:1046] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.006s
W20260807 11:10:38.057008 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 161668us
I20260807 11:10:38.061076 26033 raft_consensus.cc:2362] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3. Re-sending same reply.
W20260807 11:10:38.062831 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 43550.8us
I20260807 11:10:38.051359 25925 raft_consensus.cc:1261] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 11:10:38.071903 26936 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:37.952577 25796 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0"
dest_uuid: "606d49e053a346e486f48cb31832b815"
mode: GRACEFUL
new_leader_uuid: "b22d69db45194908ad3d345eba9badd9"
from {username='slave'} at 127.0.0.1:58114
I20260807 11:10:38.072623 25796 raft_consensus.cc:595] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Received request to transfer leadership to TS b22d69db45194908ad3d345eba9badd9
I20260807 11:10:38.073089 25796 raft_consensus.cc:603] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 11:10:38.027480 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:38.045694 25869 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:38.010216 28946 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:38.070555 29098 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.568s user 0.011s sys 0.033s
I20260807 11:10:38.091298 25869 leader_election.cc:302] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:38.057397 28647 raft_consensus.cc:686] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:38.061103 28438 raft_consensus.cc:686] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
W20260807 11:10:38.091771 26976 consensus_meta.cc:287] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.580s user 0.009s sys 0.001s
I20260807 11:10:38.113229 28423 raft_consensus.cc:2781] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:10:38.086138 29099 raft_consensus.cc:2726] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 11:10:38.056016 29207 consensus_queue.cc:1035] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20260807 11:10:38.136358 25786 raft_consensus.cc:1261] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 11:10:38.091493 26979 raft_consensus.cc:3037] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:38.093143 29194 raft_consensus.cc:2726] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 11:10:38.117556 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:38.095923 28797 consensus_queue.cc:1046] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260807 11:10:38.155190 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:38.156028 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:38.156864 25806 raft_consensus.cc:1226] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:38.156886 26039 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:38.118407 29143 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:38.119443 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:10:38.120437 29204 log.cc:924] Time spent T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.111s user 0.000s sys 0.000s
I20260807 11:10:38.121230 29052 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:38.094077 29098 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.591s user 0.011s sys 0.034s
I20260807 11:10:38.122617 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:38.125865 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:38.126044 29169 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:38.149173 29117 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:38.183651 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 77141.2us
I20260807 11:10:38.130652 25798 raft_consensus.cc:1226] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:38.134603 25799 raft_consensus.cc:1226] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:38.134914 26038 raft_consensus.cc:2454] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:38.136704 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:38.184937 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:38.060104 29037 consensus_queue.cc:1035] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.036s
I20260807 11:10:38.115082 29029 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:38.148938 25981 leader_election.cc:302] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:38.062618 29100 consensus_queue.cc:1035] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260807 11:10:38.158977 25918 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5"
dest_uuid: "4fd6381544574ddca559ce63b9243efc"
mode: GRACEFUL
new_leader_uuid: "b22d69db45194908ad3d345eba9badd9"
from {username='slave'} at 127.0.0.1:34992
I20260807 11:10:38.159716 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:38.027654 29184 consensus_queue.cc:260] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:38.163487 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:38.222998 25786 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:10:38.111642 29129 consensus_queue.cc:1046] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20260807 11:10:38.165503 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:38.167462 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:38.224691 25796 raft_consensus.cc:2379] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 3: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:38.225068 26034 raft_consensus.cc:2396] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 11:10:38.226204 26034 raft_consensus.cc:1226] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:38.228513 26034 raft_consensus.cc:1226] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:38.189512 26038 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a"
dest_uuid: "b22d69db45194908ad3d345eba9badd9"
from {username='slave'} at 127.0.0.1:59312
I20260807 11:10:38.128275 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:38.261261 26038 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:10:38.175242 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:38.261685 26038 raft_consensus.cc:3037] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:38.262251 26046 raft_consensus.cc:2379] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 3: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:38.095775 29176 consensus_queue.cc:1035] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.065s
I20260807 11:10:38.116428 29034 raft_consensus.cc:3032] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 11:10:38.263134 29034 raft_consensus.cc:729] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:38.263363 26046 raft_consensus.cc:1226] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:38.185680 26035 raft_consensus.cc:2379] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 4: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:38.155035 28423 raft_consensus.cc:686] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:38.217746 28882 raft_consensus.cc:3032] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:38.217154 28722 consensus_queue.cc:1046] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260807 11:10:38.218847 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 70025.7us
I20260807 11:10:38.222283 26041 raft_consensus.cc:1261] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:10:38.265072 28882 raft_consensus.cc:729] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:38.266047 25981 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:38.266686 26041 raft_consensus.cc:1261] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 11:10:38.266348 28882 consensus_queue.cc:260] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:38.151844 29173 raft_consensus.cc:2781] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:38.267429 25981 leader_election.cc:302] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:38.269133 29122 raft_consensus.cc:2781] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:38.269547 29122 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:38.270013 29122 raft_consensus.cc:3037] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:38.270361 29118 consensus_queue.cc:1046] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:38.271663 29206 raft_consensus.cc:3032] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:38.272073 29206 raft_consensus.cc:729] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:38.272907 29206 consensus_queue.cc:260] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:38.275290 29173 raft_consensus.cc:482] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:38.274233 26038 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:38.275682 29173 raft_consensus.cc:3037] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:38.277261 29205 raft_consensus.cc:2726] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:38.277328 28423 consensus_queue.cc:237] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:38.278475 26038 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:38.279269 29122 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:38.281134 28722 raft_consensus.cc:3032] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:38.281509 28722 raft_consensus.cc:729] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:38.281561 29122 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:38.282390 28722 consensus_queue.cc:260] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:38.283726 28722 raft_consensus.cc:3037] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:38.261675 25931 raft_consensus.cc:2379] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 5: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:38.155038 29179 consensus_queue.cc:1035] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.088s
I20260807 11:10:38.237166 25872 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:38.291622 25869 leader_election.cc:302] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
W20260807 11:10:38.239413 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 110950us
I20260807 11:10:38.126899 26976 raft_consensus.cc:2454] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:38.229379 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:38.255707 26039 raft_consensus.cc:2454] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:38.277740 25981 leader_election.cc:302] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:38.241869 25752 leader_election.cc:302] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:38.060873 29142 consensus_queue.cc:260] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:38.182725 29099 consensus_queue.cc:1046] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.116s
W20260807 11:10:38.302403 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 65379.6us
I20260807 11:10:38.302614 29235 raft_consensus.cc:2726] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 11:10:38.303718 25872 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:38.302618 29099 consensus_queue.cc:1225] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer 606d49e053a346e486f48cb31832b815 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260807 11:10:38.304800 29099 raft_consensus.cc:2726] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:38.305039 29153 raft_consensus.cc:2726] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:38.307744 26986 raft_consensus.cc:1226] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260807 11:10:38.307744 25872 leader_election.cc:302] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:38.306202 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:38.240366 29001 consensus_queue.cc:1046] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
I20260807 11:10:38.312819 26046 raft_consensus.cc:1226] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:38.287045 29029 raft_consensus.cc:2741] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260807 11:10:38.314867 26976 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 380) took 1028 ms. Trace:
W20260807 11:10:38.313485 29096 consensus_meta.cc:287] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.598s user 0.011s sys 0.000s
W20260807 11:10:38.341732 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 155476us
I20260807 11:10:38.343865 25750 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:38.342836 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 65188.5us
I20260807 11:10:38.345026 25981 leader_election.cc:302] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:38.347188 29096 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:38.351123 25798 raft_consensus.cc:1226] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:38.330616 29176 raft_consensus.cc:2781] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:10:38.362911 25808 raft_consensus.cc:3037] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
W20260807 11:10:38.365463 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 123688us
I20260807 11:10:38.367558 25917 raft_consensus.cc:1261] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 11:10:38.331367 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:38.263984 29034 consensus_queue.cc:260] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:38.369592 25981 leader_election.cc:302] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:38.325608 29122 raft_consensus.cc:2726] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:38.348907 28747 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:38.376502 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:38.257910 28438 consensus_queue.cc:237] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:38.370770 25917 raft_consensus.cc:1261] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 11:10:38.347795 29217 consensus_queue.cc:1046] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260807 11:10:38.296108 28647 consensus_queue.cc:237] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:38.412045 28964 consensus_queue.cc:1035] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20260807 11:10:38.418109 29143 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:38.364802 29176 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:38.370383 29013 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:38.326957 29098 consensus_queue.cc:1035] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.069s
I20260807 11:10:38.418778 29198 raft_consensus.cc:3032] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:38.418987 28946 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:38.375413 29006 consensus_queue.cc:1035] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.095s
I20260807 11:10:38.349771 29142 raft_consensus.cc:3037] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:38.426221 25808 raft_consensus.cc:2454] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:38.368733 29123 raft_consensus.cc:2726] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:38.391379 29127 raft_consensus.cc:2781] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:38.394731 28797 raft_consensus.cc:2781] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:38.342572 26976 rpcz_store.cc:276] 0807 11:10:37.283247 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:37.288633 (+ 5386us) service_pool.cc:227] Handling call
0807 11:10:38.311078 (+1022445us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:38.419754 25925 raft_consensus.cc:1261] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260807 11:10:38.642954 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 273450us
W20260807 11:10:38.725996 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 71255.6us
W20260807 11:10:38.743610 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 435887us
I20260807 11:10:38.523409 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:38.767737 26035 raft_consensus.cc:3037] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
W20260807 11:10:38.634140 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 253571us
I20260807 11:10:38.779717 25872 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:38.456532 29117 raft_consensus.cc:2781] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260807 11:10:38.602102 29218 log.cc:924] Time spent T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.315s user 0.000s sys 0.001s
W20260807 11:10:38.602811 29220 log.cc:924] Time spent T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.191s user 0.000s sys 0.000s
I20260807 11:10:38.571784 29122 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:38.797626 26042 consensus_meta.cc:287] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.745s user 0.012s sys 0.000s
I20260807 11:10:38.724013 29193 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:38.489549 29176 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:38.463271 29052 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:38.593748 29217 consensus_queue.cc:1046] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.223s
I20260807 11:10:38.763935 29096 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:38.648247 28722 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:38.769678 29198 raft_consensus.cc:729] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:38.587134 28747 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:38.612507 29179 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:38.644181 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:38.815930 26042 raft_consensus.cc:2396] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 11:10:38.815274 29127 raft_consensus.cc:686] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:38.770646 29099 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:38.724421 29098 consensus_queue.cc:1035] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.090s
I20260807 11:10:38.795540 29117 raft_consensus.cc:482] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:38.796881 25872 leader_election.cc:302] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:38.820393 26979 consensus_meta.cc:287] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.633s user 0.009s sys 0.002s
I20260807 11:10:38.783157 29207 consensus_queue.cc:1035] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20260807 11:10:38.873385 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:38.783304 29034 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:38.817750 29242 raft_consensus.cc:482] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:38.865876 26035 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:10:38.777733 29013 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:38.921275 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:38.918483 26042 raft_consensus.cc:2354] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4.
I20260807 11:10:38.921998 25801 raft_consensus.cc:3037] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
W20260807 11:10:38.915060 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 123231us
I20260807 11:10:38.934794 26033 raft_consensus.cc:1226] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 2. Current term is 4. Ops: []
I20260807 11:10:38.821593 29122 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:38.934953 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 591107us
I20260807 11:10:38.932186 25798 raft_consensus.cc:1261] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 11:10:38.871492 29223 raft_consensus.cc:2781] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:38.843518 29193 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:38.941474 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:38.948128 26037 raft_consensus.cc:3037] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:38.937002 25981 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:38.885378 29261 raft_consensus.cc:482] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:38.913936 29117 raft_consensus.cc:3037] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:38.910548 29196 raft_consensus.cc:482] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:38.934041 29001 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.007s sys 0.020s
W20260807 11:10:38.910610 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 117664us
I20260807 11:10:38.905936 28747 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:38.910878 29179 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:38.874403 29123 consensus_queue.cc:1046] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
I20260807 11:10:38.871145 28722 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:38.914213 26979 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:38.937024 26978 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:38.996177 28797 raft_consensus.cc:686] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 3 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:39.033298 25752 leader_election.cc:398] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Vote denied by peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009) with higher term. Message: Invalid argument: T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4.
I20260807 11:10:39.033509 26985 raft_consensus.cc:1226] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:39.033691 25752 leader_election.cc:401] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
W20260807 11:10:39.018381 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 108183us
W20260807 11:10:39.036695 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 99774.8us
I20260807 11:10:38.938696 29242 raft_consensus.cc:504] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:39.039861 28946 raft_consensus.cc:2726] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009) with higher term. Message: Invalid argument: T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4.
I20260807 11:10:39.040897 29242 leader_election.cc:288] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:39.042485 25786 raft_consensus.cc:1261] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260807 11:10:39.042626 29248 log.cc:924] Time spent T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260807 11:10:39.043746 29179 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:39.044329 29243 log.cc:924] Time spent T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.125s user 0.000s sys 0.000s
I20260807 11:10:38.917809 29127 consensus_queue.cc:237] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:39.045845 29122 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:39.046458 29122 raft_consensus.cc:2781] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:39.046855 29122 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:39.047254 29122 raft_consensus.cc:3037] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
W20260807 11:10:39.031631 29191 consensus_meta.cc:287] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 1.094s user 0.014s sys 0.000s
I20260807 11:10:39.027635 29261 raft_consensus.cc:504] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:39.029750 29254 raft_consensus.cc:2726] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 11:10:38.973491 29099 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:39.008924 29001 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.644s user 0.007s sys 0.025s
I20260807 11:10:38.985677 25750 leader_election.cc:302] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:39.116016 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 152865us
I20260807 11:10:39.116910 25750 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:39.030238 25869 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:10:39.011615 29173 consensus_meta.cc:287] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.731s user 0.011s sys 0.001s
I20260807 11:10:39.025266 28244 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:39.023831 25801 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
W20260807 11:10:39.108928 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 56187.2us
I20260807 11:10:39.119944 25981 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:39.033119 26979 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 403) took 1093 ms. Trace:
I20260807 11:10:39.122752 26037 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
W20260807 11:10:39.123299 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 72395.6us
W20260807 11:10:39.135838 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 339056us
I20260807 11:10:39.152158 25872 leader_election.cc:302] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:39.153357 25750 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:39.155962 25800 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.620s user 0.017s sys 0.003s
I20260807 11:10:39.036453 29207 consensus_queue.cc:1035] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.084s
W20260807 11:10:39.157497 25800 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.627s user 0.017s sys 0.003s
W20260807 11:10:39.161738 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 131617us
I20260807 11:10:39.164434 25917 raft_consensus.cc:1261] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 11:10:39.168305 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:39.168300 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:39.190356 25916 raft_consensus.cc:1226] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 5. Current term is 6. Ops: []
I20260807 11:10:39.194270 25981 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:39.119481 29193 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:39.410265 29250 raft_consensus.cc:482] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:39.165093 28244 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:39.411144 26975 raft_consensus.cc:1226] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: []
W20260807 11:10:39.291472 28438 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.687s user 0.016s sys 0.035s
I20260807 11:10:39.409330 29277 raft_consensus.cc:2726] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 11:10:39.313544 25920 raft_consensus.cc:1261] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 11:10:39.393805 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:39.013623 29196 raft_consensus.cc:504] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:39.124491 26979 rpcz_store.cc:276] 0807 11:10:37.939362 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:37.943039 (+ 3677us) service_pool.cc:227] Handling call
0807 11:10:39.033020 (+1089981us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:39.542953 29253 raft_consensus.cc:2781] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:39.543658 29253 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:39.544114 29253 raft_consensus.cc:3037] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
W20260807 11:10:39.412070 29251 log.cc:924] Time spent T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.257s user 0.000s sys 0.001s
W20260807 11:10:39.304543 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 116966us
I20260807 11:10:39.546262 29283 raft_consensus.cc:2726] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260807 11:10:39.312268 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 305130us
I20260807 11:10:39.304610 29179 raft_consensus.cc:2781] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:10:39.409392 29123 raft_consensus.cc:2726] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:39.450577 28722 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:39.323649 29006 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:39.124749 29173 raft_consensus.cc:504] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:39.334151 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:39.566053 29006 raft_consensus.cc:504] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:39.308883 28293 raft_consensus.cc:482] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:39.578367 28293 raft_consensus.cc:504] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:39.579921 29006 leader_election.cc:288] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:39.453456 29099 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:39.193369 28946 raft_consensus.cc:3037] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:39.409814 29244 raft_consensus.cc:3037] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:39.453750 28747 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:39.375272 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 181110us
I20260807 11:10:39.452994 29282 consensus_queue.cc:1035] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260807 11:10:39.334776 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 176422us
I20260807 11:10:39.484444 25917 raft_consensus.cc:1261] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:10:39.314452 28797 consensus_queue.cc:237] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:39.507479 29013 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:39.524293 29295 raft_consensus.cc:2781] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:39.524443 25921 raft_consensus.cc:3032] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:39.537616 26984 raft_consensus.cc:3037] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:39.339231 26976 raft_consensus.cc:1226] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:39.545955 26985 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50942 (request call id 382) took 1248 ms. Trace:
I20260807 11:10:39.546936 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1598) took 1458 ms. Trace:
I20260807 11:10:39.548918 25750 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc
W20260807 11:10:39.537573 28438 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.927s user 0.025s sys 0.042s
I20260807 11:10:39.566566 29196 leader_election.cc:288] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:39.586911 26042 raft_consensus.cc:1261] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 11:10:39.607573 26033 rpcz_store.cc:276] 0807 11:10:38.088526 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:38.222367 (+133841us) service_pool.cc:227] Handling call
0807 11:10:38.934597 (+712230us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:38.936953 (+ 2356us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:39.546742 (+609789us) spinlock_profiling.cc:243] Waited 676 ms on lock 0x7b54000bb150. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:39.546875 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1554376448}
I20260807 11:10:39.595219 25981 leader_election.cc:302] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:39.606552 26985 rpcz_store.cc:276] 0807 11:10:38.296998 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:38.300872 (+ 3874us) service_pool.cc:227] Handling call
0807 11:10:39.031317 (+730445us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:39.033943 (+ 2626us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:39.545669 (+511726us) spinlock_profiling.cc:243] Waited 701 ms on lock 0x7b540074f9d0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:39.545850 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1611578112}
I20260807 11:10:39.603961 25921 raft_consensus.cc:729] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
W20260807 11:10:39.594437 29117 consensus_meta.cc:287] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.577s user 0.009s sys 0.001s
I20260807 11:10:39.599841 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1692) took 1363 ms. Trace:
I20260807 11:10:39.621349 26046 rpcz_store.cc:276] 0807 11:10:38.236625 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:38.280444 (+ 43819us) service_pool.cc:227] Handling call
0807 11:10:38.301937 (+ 21493us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:38.327918 (+ 25981us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:39.599646 (+1271728us) spinlock_profiling.cc:243] Waited 20.4 ms on lock 0x7b54001c5050. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:39.599780 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":48544896}
I20260807 11:10:39.580541 28293 leader_election.cc:288] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:39.626420 29292 log.cc:924] Time spent T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.182s user 0.000s sys 0.001s
W20260807 11:10:39.605068 29034 consensus_meta.cc:287] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.683s user 0.009s sys 0.002s
W20260807 11:10:39.603425 29238 log.cc:924] Time spent T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.144s user 0.000s sys 0.001s
I20260807 11:10:39.586246 29179 raft_consensus.cc:686] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 5 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:39.587921 29123 raft_consensus.cc:482] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:39.600179 28761 raft_consensus.cc:482] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:39.582063 29260 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
W20260807 11:10:39.674436 29176 consensus_meta.cc:287] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.823s user 0.011s sys 0.000s
I20260807 11:10:39.498044 29250 raft_consensus.cc:504] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:39.677618 29206 raft_consensus.cc:3037] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:39.677412 29117 raft_consensus.cc:504] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:39.877400 29117 leader_election.cc:288] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:39.682451 29111 raft_consensus.cc:2781] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election won for term 2
W20260807 11:10:39.750216 25872 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.005s sys 0.022s
I20260807 11:10:39.878252 29111 raft_consensus.cc:686] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 2 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
W20260807 11:10:39.878479 25872 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.715s user 0.005s sys 0.022s
I20260807 11:10:39.879462 29111 consensus_queue.cc:237] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:39.603885 29295 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:39.742615 28449 raft_consensus.cc:3032] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Stepping down as leader of term 3
W20260807 11:10:39.743360 29290 log.cc:924] Time spent T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.207s user 0.000s sys 0.000s
I20260807 11:10:39.881065 29295 raft_consensus.cc:3037] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:39.881152 28449 raft_consensus.cc:729] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
W20260807 11:10:39.713372 29084 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.049s sys 0.046s
W20260807 11:10:39.882278 29084 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.710s user 0.049s sys 0.047s
I20260807 11:10:39.882620 28449 consensus_queue.cc:260] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:39.884451 29296 log.cc:924] Time spent T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.334s user 0.000s sys 0.000s
I20260807 11:10:39.743955 29307 raft_consensus.cc:482] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:39.647994 28747 raft_consensus.cc:2781] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:39.889184 28747 raft_consensus.cc:482] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:39.889112 29307 raft_consensus.cc:504] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:39.889963 28747 raft_consensus.cc:3037] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:39.616940 29013 consensus_queue.cc:1035] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.144s
I20260807 11:10:39.665009 29127 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:39.893769 29127 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:39.895051 29295 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:39.896381 26037 raft_consensus.cc:1226] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:39.898270 29295 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:39.745100 29207 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:39.925508 29273 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:39.926299 29273 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:39.929373 29273 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:39.971894 29301 log.cc:924] Time spent T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20260807 11:10:39.979969 26046 raft_consensus.cc:3037] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:39.980321 25918 raft_consensus.cc:595] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Received request to transfer leadership to TS b22d69db45194908ad3d345eba9badd9
W20260807 11:10:39.980713 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 385593us
W20260807 11:10:39.981248 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 829078us
I20260807 11:10:39.981829 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:39.980753 26936 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:39.891839 29307 leader_election.cc:288] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:39.983542 25803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1627) took 1044 ms. Trace:
I20260807 11:10:39.989068 25799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 279) took 1633 ms. Trace:
I20260807 11:10:39.989279 25917 raft_consensus.cc:1226] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260807 11:10:39.980320 25921 consensus_queue.cc:260] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:39.990353 25920 raft_consensus.cc:2354] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4.
I20260807 11:10:39.993937 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:39.994899 25796 raft_consensus.cc:2454] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:10:39.996817 25796 raft_consensus.cc:1226] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:40.000564 25872 leader_election.cc:302] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:39.989493 25799 rpcz_store.cc:276] 0807 11:10:38.346807 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:38.346988 (+ 181us) service_pool.cc:227] Handling call
0807 11:10:38.347355 (+ 367us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:38.412253 (+ 64898us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:38.412288 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:38.421045 (+ 8757us) log.cc:842] Serialized 26 byte log entry
0807 11:10:38.855596 (+434551us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:10:38.855665 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:38.855767 (+ 102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:39.546025 (+690258us) raft_consensus.cc:1671] finished
0807 11:10:39.546056 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:39.980422 (+434366us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46978,"raft.queue_time_us":283986,"spinlock_wait_cycles":16685952,"thread_start_us":434017,"threads_started":1,"wal-append.queue_time_us":609008}
I20260807 11:10:39.990136 25800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1626) took 1591 ms. Trace:
I20260807 11:10:40.006260 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:40.005635 25921 raft_consensus.cc:3037] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:40.007026 25919 raft_consensus.cc:3037] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:40.009271 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:40.008638 25981 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:39.986498 25803 rpcz_store.cc:276] 0807 11:10:38.939333 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:38.939618 (+ 285us) service_pool.cc:227] Handling call
0807 11:10:38.974220 (+ 34602us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:38.996645 (+ 22425us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:38.996697 (+ 52us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:39.017449 (+ 20752us) log.cc:842] Serialized 26 byte log entry
0807 11:10:39.134996 (+117547us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:39.135031 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:39.135100 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:39.669376 (+534276us) raft_consensus.cc:1671] finished
0807 11:10:39.669403 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:39.983478 (+314075us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3992,"raft.queue_time_us":124,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":27571,"thread_start_us":93401,"threads_started":1,"wal-append.queue_time_us":280231}
I20260807 11:10:39.867517 28761 raft_consensus.cc:504] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:40.006142 25800 rpcz_store.cc:276] 0807 11:10:38.392639 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:38.392830 (+ 191us) service_pool.cc:227] Handling call
0807 11:10:38.415418 (+ 22588us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:38.415635 (+ 217us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:38.415668 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:38.416073 (+ 405us) log.cc:842] Serialized 26 byte log entry
0807 11:10:39.169790 (+753717us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:39.169831 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:39.169897 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:39.984083 (+814186us) raft_consensus.cc:1671] finished
0807 11:10:39.984105 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:39.984244 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":639247,"threads_started":1,"wal-append.queue_time_us":983002}
I20260807 11:10:40.028096 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:40.030642 25872 leader_election.cc:302] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:40.031713 25926 raft_consensus.cc:1226] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:39.763083 29123 raft_consensus.cc:504] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:40.038424 25872 leader_election.cc:302] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:40.039108 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:40.002362 29254 raft_consensus.cc:482] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:40.041067 26033 raft_consensus.cc:1226] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:40.004492 29084 raft_consensus.cc:2781] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:39.871739 29176 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:40.019878 29307 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:40.051682 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:40.017932 28244 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:40.055905 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 507042us
I20260807 11:10:39.870616 29260 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:40.025429 29268 raft_consensus.cc:2726] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 11:10:40.047490 29127 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:39.943401 29207 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:40.051112 29295 raft_consensus.cc:2726] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:40.076682 25917 raft_consensus.cc:1226] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260807 11:10:40.081900 25919 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:40.082670 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:40.062958 29084 raft_consensus.cc:686] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:40.099813 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:40.013936 29179 consensus_queue.cc:237] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:40.106355 25921 raft_consensus.cc:1261] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 11:10:40.108121 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:40.109880 25919 raft_consensus.cc:2454] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:40.068305 28567 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:40.056448 29013 consensus_queue.cc:1046] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.030s
I20260807 11:10:40.074694 29319 raft_consensus.cc:482] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:40.097536 29266 raft_consensus.cc:3032] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:40.096516 29302 raft_consensus.cc:482] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:40.068410 29315 raft_consensus.cc:482] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:40.077184 29273 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:40.118108 26984 consensus_meta.cc:287] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.512s user 0.010s sys 0.000s
I20260807 11:10:40.119239 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:40.068269 29307 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:40.115044 26046 raft_consensus.cc:1261] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260807 11:10:40.076288 29119 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:40.120733 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:40.150096 26984 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:40.084951 28244 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:40.162034 26046 raft_consensus.cc:1226] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:40.151758 25917 raft_consensus.cc:2454] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:40.118739 29295 consensus_queue.cc:1046] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20260807 11:10:40.094210 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:40.155542 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 348) took 1678 ms. Trace:
W20260807 11:10:40.155778 29244 consensus_meta.cc:287] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.551s user 0.009s sys 0.000s
I20260807 11:10:40.149411 29279 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:40.205281 29279 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:40.134176 29302 raft_consensus.cc:504] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:40.207722 29279 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:40.208379 25920 raft_consensus.cc:2454] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:10:40.158311 29123 leader_election.cc:288] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:40.124697 29127 raft_consensus.cc:482] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:40.093262 28797 raft_consensus.cc:2726] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:40.125316 29266 raft_consensus.cc:729] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:40.056980 29254 raft_consensus.cc:504] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:40.124307 28567 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:40.141326 29303 raft_consensus.cc:482] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:40.133213 28293 raft_consensus.cc:482] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:40.189675 29244 raft_consensus.cc:2726] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:40.142823 29223 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:40.237025 26046 raft_consensus.cc:1261] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 11:10:40.175457 26034 rpcz_store.cc:276] 0807 11:10:38.477649 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:38.477938 (+ 289us) service_pool.cc:227] Handling call
0807 11:10:38.787367 (+309429us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:39.281518 (+494151us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:39.282667 (+ 1149us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:39.309000 (+ 26333us) log.cc:842] Serialized 26 byte log entry
0807 11:10:39.609337 (+300337us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:39.609375 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:39.609447 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:40.155308 (+545861us) raft_consensus.cc:1671] finished
0807 11:10:40.155331 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:40.155476 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":28095,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":24463,"thread_start_us":605988,"threads_started":2,"wal-append.queue_time_us":741716}
W20260807 11:10:40.257378 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 219486us
W20260807 11:10:40.250135 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 205178us
I20260807 11:10:40.141525 29310 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:40.150964 29273 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:40.208469 29302 leader_election.cc:288] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:40.163599 29318 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:40.282557 29250 leader_election.cc:288] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:40.251327 26976 raft_consensus.cc:2454] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:40.298419 25981 leader_election.cc:302] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:40.296232 28293 raft_consensus.cc:504] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:40.295537 28761 leader_election.cc:288] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:40.178890 28244 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:40.263513 29305 consensus_queue.cc:1035] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.112s
I20260807 11:10:40.163522 29119 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:40.285547 29127 raft_consensus.cc:504] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:40.146095 29315 raft_consensus.cc:504] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:40.130823 29319 raft_consensus.cc:504] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:40.145108 29084 consensus_queue.cc:237] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:40.322646 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:40.296708 29223 raft_consensus.cc:3037] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:40.344812 26046 raft_consensus.cc:2454] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:10:40.307423 25872 leader_election.cc:302] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:40.348124 26938 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:40.319675 28567 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:40.360320 29254 leader_election.cc:288] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:40.361433 29127 leader_election.cc:288] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:40.362157 29247 raft_consensus.cc:2781] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:10:40.374665 25872 leader_election.cc:302] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
W20260807 11:10:40.363536 26978 consensus_meta.cc:287] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 1.294s user 0.011s sys 0.000s
I20260807 11:10:40.296452 29303 raft_consensus.cc:504] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:40.302767 29314 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:40.366077 28293 leader_election.cc:288] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:40.366276 28946 consensus_meta.cc:287] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.785s user 0.011s sys 0.000s
I20260807 11:10:40.303411 29295 consensus_queue.cc:1046] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.063s
I20260807 11:10:40.369684 29260 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:40.320719 29001 consensus_queue.cc:1035] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
W20260807 11:10:40.319113 29122 consensus_meta.cc:287] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 1.271s user 0.014s sys 0.000s
I20260807 11:10:40.325470 29302 raft_consensus.cc:2726] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:40.393931 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:40.394727 26034 raft_consensus.cc:2454] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:40.321456 29311 raft_consensus.cc:2726] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 11:10:40.343942 29244 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:40.349311 29207 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:40.377512 25981 leader_election.cc:302] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:40.378998 25872 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:40.301223 29034 consensus_queue.cc:1035] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.074s
I20260807 11:10:40.306632 29176 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:40.382362 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:40.385514 26041 raft_consensus.cc:1226] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4. Ops: []
W20260807 11:10:40.305240 29142 consensus_meta.cc:287] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 1.541s user 0.012s sys 0.003s
I20260807 11:10:40.374049 29247 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:40.390003 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:40.436816 26035 raft_consensus.cc:2454] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:40.435092 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:40.437987 26041 raft_consensus.cc:2379] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 6: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:40.319439 28797 consensus_queue.cc:1046] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.061s
I20260807 11:10:40.390918 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:40.438995 26033 raft_consensus.cc:2379] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 2: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:40.392970 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:40.376657 26978 raft_consensus.cc:2454] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:10:40.395787 26042 raft_consensus.cc:1226] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:40.381467 29254 raft_consensus.cc:2781] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election won for term 5
W20260807 11:10:40.399760 29299 log.cc:924] Time spent T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.657s user 0.002s sys 0.008s
I20260807 11:10:40.405133 25981 leader_election.cc:302] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:40.402091 29328 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:40.410085 25872 leader_election.cc:302] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:40.410409 28747 consensus_meta.cc:287] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.520s user 0.010s sys 0.001s
W20260807 11:10:40.410801 29333 log.cc:924] Time spent T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.085s user 0.000s sys 0.001s
I20260807 11:10:40.411213 29330 raft_consensus.cc:2726] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260807 11:10:40.411825 29253 consensus_meta.cc:287] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.867s user 0.006s sys 0.006s
I20260807 11:10:40.461436 29253 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:40.389524 28946 raft_consensus.cc:2726] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:40.417369 26046 raft_consensus.cc:2454] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:40.299561 29310 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:40.434955 25752 leader_election.cc:302] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:40.468189 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:40.468901 26042 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:40.435326 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:40.471122 26034 raft_consensus.cc:2454] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:40.438078 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:40.439626 26036 raft_consensus.cc:2454] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:40.441201 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:40.393918 29122 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:40.444581 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 321090us
I20260807 11:10:40.444610 26987 raft_consensus.cc:1226] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 5. Current term is 6. Ops: []
W20260807 11:10:40.449820 29206 consensus_meta.cc:287] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.532s user 0.008s sys 0.003s
I20260807 11:10:40.450891 28797 raft_consensus.cc:3032] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:40.451874 29119 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:40.453043 25808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1481) took 2644 ms (client timeout 1565 ms). Trace:
I20260807 11:10:40.463480 29254 raft_consensus.cc:686] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 5 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:40.396380 29273 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:40.405390 29307 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:40.385152 29127 consensus_queue.cc:1035] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260807 11:10:40.304513 29318 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:40.457661 29327 log.cc:924] Time spent T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.160s user 0.001s sys 0.000s
W20260807 11:10:40.457908 29329 log.cc:924] Time spent T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.121s user 0.000s sys 0.001s
I20260807 11:10:40.458647 28747 raft_consensus.cc:504] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:40.321624 29266 consensus_queue.cc:260] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:40.459800 26978 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 385) took 1826 ms (client timeout 1518 ms). Trace:
I20260807 11:10:40.463943 26041 raft_consensus.cc:1261] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:10:40.438784 29193 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:40.384567 29323 consensus_queue.cc:1046] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.225s
I20260807 11:10:40.439028 29247 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:40.411585 29244 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:40.475445 26033 raft_consensus.cc:2454] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:40.384040 29314 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:40.494707 28946 consensus_queue.cc:1046] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
I20260807 11:10:40.412973 29302 raft_consensus.cc:3032] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:40.498258 29119 consensus_queue.cc:1035] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.005s
W20260807 11:10:40.498984 25808 rpcz_store.cc:269] 0807 11:10:37.808523 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:37.808640 (+ 117us) service_pool.cc:227] Handling call
0807 11:10:40.452827 (+2644187us) spinlock_profiling.cc:243] Waited 497 ms on lock 0x7b54002d3cd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:40.452947 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1142388096}
I20260807 11:10:40.466116 29328 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:40.506413 26046 raft_consensus.cc:1226] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: []
W20260807 11:10:40.511051 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 105953us
I20260807 11:10:40.408550 29334 consensus_queue.cc:1046] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.099s
W20260807 11:10:40.611132 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 201115us
I20260807 11:10:40.479363 29094 raft_consensus.cc:2781] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:40.645749 25750 leader_election.cc:398] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Vote denied by peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465) with higher term. Message: Invalid argument: T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4.
W20260807 11:10:40.728653 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 83088.5us
I20260807 11:10:40.727974 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:40.645195 29235 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260807 11:10:40.485989 29176 raft_consensus.cc:2726] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 11:10:40.503311 29117 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:40.498687 28797 raft_consensus.cc:729] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:40.749626 25750 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:40.752038 26042 raft_consensus.cc:1226] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:40.595440 29311 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:40.686025 29344 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:40.692312 29179 raft_consensus.cc:482] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:40.660072 29173 leader_election.cc:288] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:40.548653 29001 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:40.706048 29315 leader_election.cc:288] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:40.639007 28882 raft_consensus.cc:3037] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:40.602524 29319 leader_election.cc:288] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:40.610244 29253 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:40.677143 29303 leader_election.cc:288] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:40.753968 29266 raft_consensus.cc:3037] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:40.754118 29334 consensus_queue.cc:1046] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
I20260807 11:10:40.775876 29321 raft_consensus.cc:2781] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:40.756474 29117 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:40.805589 29127 consensus_queue.cc:1035] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:10:40.806782 29034 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:40.807925 29340 consensus_queue.cc:1046] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:10:40.729827 29193 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:40.713202 26978 rpcz_store.cc:269] 0807 11:10:38.627312 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:38.628704 (+ 1392us) service_pool.cc:227] Handling call
0807 11:10:40.453656 (+1824952us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:40.846177 29340 consensus_queue.cc:1225] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer 4fd6381544574ddca559ce63b9243efc log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260807 11:10:40.846709 29328 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:40.846901 29340 raft_consensus.cc:3032] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:40.847426 29340 raft_consensus.cc:729] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:40.808882 29311 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:40.643296 29260 consensus_queue.cc:1035] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.167s
I20260807 11:10:40.857080 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:40.809793 29354 log.cc:924] Time spent T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260807 11:10:40.670636 29336 raft_consensus.cc:2781] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:40.858337 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:40.858654 29336 raft_consensus.cc:482] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:40.859045 29336 raft_consensus.cc:3037] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:40.859629 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:40.810809 28747 leader_election.cc:288] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:40.811702 29310 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:40.748560 29330 consensus_queue.cc:1046] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
I20260807 11:10:40.861841 29330 consensus_queue.cc:1225] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Peer b22d69db45194908ad3d345eba9badd9 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260807 11:10:40.617476 29254 consensus_queue.cc:237] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:40.839515 29001 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:40.812867 29314 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:40.644730 29122 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:40.747438 29094 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:40.865536 29094 raft_consensus.cc:3037] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:40.858349 29311 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:40.865864 29244 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:40.847453 29034 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:40.860750 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:40.868276 29034 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:40.868512 26033 raft_consensus.cc:2379] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 4: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:40.523316 29206 raft_consensus.cc:2781] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:40.870725 29206 raft_consensus.cc:482] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:40.871131 29206 raft_consensus.cc:3037] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:40.820998 25981 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:10:40.826814 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 67888.5us
W20260807 11:10:40.873162 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 52383.8us
I20260807 11:10:40.875275 29122 raft_consensus.cc:2781] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:40.875783 29122 raft_consensus.cc:686] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:40.876693 29122 consensus_queue.cc:237] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:40.838709 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 81643.8us
I20260807 11:10:40.712055 29302 raft_consensus.cc:729] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:40.840957 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 228) took 1675 ms. Trace:
I20260807 11:10:40.811733 28244 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:40.842795 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:40.816884 28946 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:40.880301 25750 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:40.643139 29223 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:40.850055 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:40.854471 29340 consensus_queue.cc:260] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:40.838181 29179 raft_consensus.cc:504] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:40.811336 29318 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:40.879711 26033 raft_consensus.cc:1226] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:40.864393 29001 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:40.883509 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:40.884059 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:40.872578 29094 raft_consensus.cc:504] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:40.881234 25932 rpcz_store.cc:276] 0807 11:10:39.163042 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:39.163201 (+ 159us) service_pool.cc:227] Handling call
0807 11:10:39.291078 (+127877us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:39.523919 (+232841us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:39.523962 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:39.524465 (+ 503us) log.cc:842] Serialized 26 byte log entry
0807 11:10:40.069862 (+545397us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:10:40.069891 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:40.069943 (+ 52us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:40.838554 (+768611us) raft_consensus.cc:1671] finished
0807 11:10:40.838575 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:40.838704 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":301632,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":178,"thread_start_us":283925,"threads_started":2,"wal-append.queue_time_us":639098}
W20260807 11:10:40.954255 25981 connection.cc:232] RPC call timeout handler was delayed by -1.9720548330806196s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1565s for a response
W20260807 11:10:40.959969 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 79779.3us
I20260807 11:10:40.886210 25800 raft_consensus.cc:1226] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:40.960445 25808 raft_consensus.cc:3037] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
W20260807 11:10:40.959533 25981 connection.cc:232] RPC call timeout handler was delayed by -1.8991548330523074s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1646s for a response
W20260807 11:10:40.962425 25981 connection.cc:232] RPC call timeout handler was delayed by -1.6570548329036683s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1915s for a response
W20260807 11:10:40.962723 25981 connection.cc:232] RPC call timeout handler was delayed by -1.5319548330735415s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2054s for a response
W20260807 11:10:40.962992 25981 connection.cc:232] RPC call timeout handler was delayed by -1.5040548329707235s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20850000000000002s for a response
W20260807 11:10:40.963266 25981 connection.cc:232] RPC call timeout handler was delayed by -1.5040548329707235s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20850000000000002s for a response
W20260807 11:10:40.963616 25981 connection.cc:232] RPC call timeout handler was delayed by -1.3735548330005258s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.223s for a response
I20260807 11:10:40.963833 26041 raft_consensus.cc:2379] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 5: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:40.964991 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:40.965868 26986 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50960 (request call id 332) took 3027 ms. Trace:
I20260807 11:10:40.966399 25750 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:40.966709 25800 raft_consensus.cc:1226] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:40.969072 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:40.969820 25799 raft_consensus.cc:2379] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 5: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:40.969827 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:40.966265 26986 rpcz_store.cc:276] 0807 11:10:37.930953 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:37.931076 (+ 123us) service_pool.cc:227] Handling call
0807 11:10:38.300723 (+369647us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:38.311573 (+ 10850us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:40.112215 (+1800642us) spinlock_profiling.cc:243] Waited 6.8 ms on lock 0x7b54005dded4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:40.958612 (+846397us) spinlock_profiling.cc:243] Waited 369 ms on lock 0x7b54005dded0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:40.958752 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":864983296}
I20260807 11:10:40.974354 29094 leader_election.cc:288] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:40.975039 28964 raft_consensus.cc:2781] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:40.975450 28964 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:40.975792 28964 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:40.967257 29318 raft_consensus.cc:2781] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:40.964956 28747 consensus_queue.cc:1035] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.013s
I20260807 11:10:40.978993 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:40.979127 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:40.979671 26985 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:40.980132 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:40.978467 29179 leader_election.cc:288] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:40.949198 28797 consensus_queue.cc:260] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:40.935631 29034 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:40.981093 29340 raft_consensus.cc:3037] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:40.981444 29034 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:40.983549 29034 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:40.984112 25796 raft_consensus.cc:1226] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:40.829555 29344 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:41.001533 25981 leader_election.cc:302] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:40.984048 26984 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:41.003585 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:41.004578 29094 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:40.957142 29231 raft_consensus.cc:3032] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:41.009608 26034 raft_consensus.cc:1261] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:10:41.010386 25803 raft_consensus.cc:2354] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5.
I20260807 11:10:41.005726 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:41.023245 26036 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:10:41.018815 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:41.005399 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:41.010746 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:41.024497 25786 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:40.880833 29321 raft_consensus.cc:686] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:41.013418 25808 raft_consensus.cc:2454] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:40.972365 29307 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:41.025087 25798 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:10:40.977108 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:40.977120 29318 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:41.038304 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:40.999866 29127 raft_consensus.cc:482] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:41.043212 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:41.004233 29328 raft_consensus.cc:482] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:41.004787 28882 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:41.018165 29173 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:41.032378 29214 log.cc:924] Time spent T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 2.812s user 0.013s sys 0.016s
I20260807 11:10:41.009419 29244 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:41.017283 28964 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:40.999718 29247 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:41.021999 29094 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:41.000205 29302 consensus_queue.cc:260] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:41.023138 29231 raft_consensus.cc:729] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:41.001248 28244 consensus_queue.cc:1046] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260807 11:10:40.972213 28946 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:41.041843 29307 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:41.009306 29352 raft_consensus.cc:2781] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:41.055576 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
W20260807 11:10:41.289847 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 284623us
I20260807 11:10:41.460489 25872 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:41.056687 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:41.055971 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:41.461145 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:41.057231 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:41.466403 25981 leader_election.cc:302] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:41.306758 25798 raft_consensus.cc:2454] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 5.
W20260807 11:10:41.408866 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 403160us
I20260807 11:10:41.031584 29179 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:41.171340 25916 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1619) took 1696 ms. Trace:
I20260807 11:10:41.487836 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:41.487965 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:41.362263 25918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34992 (request call id 11) took 3225 ms. Trace:
I20260807 11:10:41.465010 25803 raft_consensus.cc:2454] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:41.476811 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:41.056367 26036 raft_consensus.cc:1261] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260807 11:10:41.484912 25981 leader_election.cc:398] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195) with higher term. Message: Invalid argument: T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5.
I20260807 11:10:41.386278 25786 raft_consensus.cc:2454] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 6.
I20260807 11:10:41.461040 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:41.031366 29001 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:41.476752 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:41.500571 25796 raft_consensus.cc:2454] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:41.501819 26033 raft_consensus.cc:2396] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 11:10:41.502521 26036 pending_rounds.cc:76] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9: Aborting all ops after (but not including) 2
I20260807 11:10:41.044126 29318 raft_consensus.cc:3037] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:41.502297 25981 leader_election.cc:401] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260807 11:10:41.501487 25918 rpcz_store.cc:276] 0807 11:10:38.136563 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:38.137318 (+ 755us) service_pool.cc:227] Handling call
0807 11:10:41.362041 (+3224723us) spinlock_profiling.cc:243] Waited 1.76 s on lock 0x7b54003629d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6bebc3 00007f3fc460881a 00007f3fbdcfbd04 00007f3fbdcfbc92 00007f3fbdcfbbc1 00007f3fbdcfbb3c 00007f3fbdcfadf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:41.362155 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4044248960}
I20260807 11:10:41.503211 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:41.506762 26986 raft_consensus.cc:2354] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7.
I20260807 11:10:41.486996 25916 rpcz_store.cc:276] 0807 11:10:39.475281 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:39.569749 (+ 94468us) service_pool.cc:227] Handling call
0807 11:10:41.170611 (+1600862us) spinlock_profiling.cc:243] Waited 2.18 ms on lock 0x7b58002e0e98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc461a2b0 00007f3fc46057f9 00007f3fc46052a2 00007f3fbdceb684
0807 11:10:41.170837 (+ 226us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:41.171020 (+ 183us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:41.171047 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:41.171089 (+ 42us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:41.171122 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:41.171178 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:41.171286 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5024128}
W20260807 11:10:41.309326 29297 log.cc:924] Time spent T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 1.915s user 0.001s sys 0.004s
W20260807 11:10:41.318807 29343 log.cc:924] Time spent T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.619s user 0.001s sys 0.000s
I20260807 11:10:41.502846 26036 pending_rounds.cc:97] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260807 11:10:41.044890 29357 raft_consensus.cc:482] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:41.062539 29223 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:41.058403 29344 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:41.166540 28882 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:41.167187 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:41.542057 25797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 283) took 2619 ms. Trace:
I20260807 11:10:41.543937 28882 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:41.542562 25797 rpcz_store.cc:276] 0807 11:10:38.922524 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:38.938943 (+ 16419us) service_pool.cc:227] Handling call
0807 11:10:38.978050 (+ 39107us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:38.978444 (+ 394us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:38.978475 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:39.003784 (+ 25309us) log.cc:842] Serialized 26 byte log entry
0807 11:10:39.134953 (+131169us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:39.135015 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:39.135100 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:41.308663 (+2173563us) spinlock_profiling.cc:243] Waited 15.7 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:10:41.541644 (+232981us) raft_consensus.cc:1671] finished
0807 11:10:41.541675 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:41.541960 (+ 285us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8209,"raft.queue_time_us":17757,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":3637,"spinlock_wait_cycles":36039040,"thread_start_us":101685,"threads_started":1,"wal-append.queue_time_us":272859}
I20260807 11:10:41.504698 29001 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:41.547879 29001 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:41.511970 29347 raft_consensus.cc:3037] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:41.554471 29094 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:41.506095 29352 raft_consensus.cc:686] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
W20260807 11:10:41.560895 29266 consensus_meta.cc:287] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.720s user 0.012s sys 0.000s
I20260807 11:10:41.529285 26036 raft_consensus.cc:2866] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:10:41.484992 29260 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:41.483237 29276 raft_consensus.cc:2726] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:41.487274 29358 consensus_queue.cc:1046] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.052s
I20260807 11:10:41.539381 29223 raft_consensus.cc:482] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:41.502048 29118 consensus_queue.cc:1046] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20260807 11:10:41.484612 29244 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:41.538198 28244 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:41.562702 26034 raft_consensus.cc:1226] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:41.482844 28449 raft_consensus.cc:3037] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:41.512385 25981 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:41.500895 29029 raft_consensus.cc:2781] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:10:41.513782 29250 raft_consensus.cc:2726] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 11:10:41.167459 29206 raft_consensus.cc:504] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:41.567607 29260 raft_consensus.cc:504] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:41.567988 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 347) took 3112 ms. Trace:
I20260807 11:10:41.571087 29184 raft_consensus.cc:3037] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:41.575534 29198 consensus_queue.cc:260] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:41.575841 29231 consensus_queue.cc:260] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:41.564496 29357 raft_consensus.cc:504] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:41.172464 29321 consensus_queue.cc:237] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:41.619817 29373 raft_consensus.cc:3032] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:41.622964 29373 raft_consensus.cc:729] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:41.623833 29260 leader_election.cc:288] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:41.170922 29336 raft_consensus.cc:504] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:41.628245 29336 leader_election.cc:288] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:41.063179 29328 raft_consensus.cc:504] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:41.629140 29029 raft_consensus.cc:686] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 2 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:41.630328 29328 leader_election.cc:288] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:41.630156 29029 consensus_queue.cc:237] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:41.050371 29127 raft_consensus.cc:504] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:41.631763 29123 raft_consensus.cc:3032] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:41.632174 29123 raft_consensus.cc:729] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:41.632783 29127 leader_election.cc:288] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:41.632988 29123 consensus_queue.cc:260] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:41.633818 25806 raft_consensus.cc:2396] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 11:10:41.634358 29123 raft_consensus.cc:3037] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:41.636041 29357 leader_election.cc:288] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:41.636663 29231 raft_consensus.cc:3037] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:41.604590 29170 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:41.620169 29266 raft_consensus.cc:2781] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:10:41.641292 29266 raft_consensus.cc:686] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 5 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:41.586211 29247 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:41.558688 29307 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:41.589547 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:41.642640 26035 raft_consensus.cc:2454] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
I20260807 11:10:41.643386 29198 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:41.643638 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:41.595124 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:41.645306 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:41.645910 25918 raft_consensus.cc:3037] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:41.601069 29310 consensus_queue.cc:1046] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:10:41.648888 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1716) took 1260 ms. Trace:
I20260807 11:10:41.617681 29223 raft_consensus.cc:504] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:41.649446 29310 raft_consensus.cc:3032] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:41.649272 26042 rpcz_store.cc:276] 0807 11:10:40.388679 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:40.469560 (+ 80881us) service_pool.cc:227] Handling call
0807 11:10:40.496057 (+ 26497us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:40.755357 (+259300us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:41.648644 (+893287us) spinlock_profiling.cc:243] Waited 253 ms on lock 0x7b540022b854. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:41.648832 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":581038336}
I20260807 11:10:41.650046 29310 raft_consensus.cc:729] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:41.650736 29223 leader_election.cc:288] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:41.651062 29310 consensus_queue.cc:260] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:41.640483 29170 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:41.601696 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:41.653640 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:41.653978 29170 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:41.654951 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:41.656308 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:41.172680 29254 consensus_queue.cc:1046] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.500s
I20260807 11:10:41.608042 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:41.669337 25921 raft_consensus.cc:2454] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
W20260807 11:10:41.609192 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 97066.4us
I20260807 11:10:41.500667 29330 consensus_queue.cc:1035] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.094s
I20260807 11:10:41.613226 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:41.670819 25750 leader_election.cc:302] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:41.503625 29179 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:41.497089 29173 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:41.575567 28964 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:41.617036 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:41.618124 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:41.620533 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:41.621703 25932 raft_consensus.cc:1261] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:10:41.630908 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 350) took 3048 ms. Trace:
I20260807 11:10:41.693495 26038 rpcz_store.cc:276] 0807 11:10:38.582743 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:38.582913 (+ 170us) service_pool.cc:227] Handling call
0807 11:10:41.454627 (+2871714us) spinlock_profiling.cc:243] Waited 16.6 ms on lock 0x7b5800270998. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc461a2b0 00007f3fc46057f9 00007f3fc46052a2 00007f3fbdceb684
0807 11:10:41.630109 (+175482us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b5800270998. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764b7c 00007f3fbc76473b 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc461a2b0 00007f3fc46057f9 00007f3fc46052a2 00007f3fbdceb684
0807 11:10:41.630291 (+ 182us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:41.630473 (+ 182us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:41.630503 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:41.630646 (+ 143us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:10:41.630683 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:41.630741 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:41.630852 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":73028864}
W20260807 11:10:41.581475 25920 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35276 (request call id 229) took 1553 ms (client timeout 1848 ms). Trace:
W20260807 11:10:41.695088 25920 rpcz_store.cc:269] 0807 11:10:40.028185 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:40.028505 (+ 320us) service_pool.cc:227] Handling call
0807 11:10:41.581233 (+1552728us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b5400408b54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:41.581375 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":342918912}
I20260807 11:10:41.696630 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:41.698100 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:41.698772 25920 raft_consensus.cc:3037] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:41.700451 26976 raft_consensus.cc:2454] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:41.701854 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:41.692201 25919 raft_consensus.cc:2454] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:41.691519 25917 raft_consensus.cc:2454] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:41.642230 29266 consensus_queue.cc:237] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:41.648468 29206 leader_election.cc:288] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:41.650663 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:41.659030 29303 consensus_queue.cc:1046] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 11:10:41.662014 29310 raft_consensus.cc:3037] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:41.664623 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:41.708482 26986 raft_consensus.cc:2379] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 2: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:41.667341 29342 raft_consensus.cc:3032] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 11:10:41.667706 29193 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:41.667358 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:41.623842 29373 consensus_queue.cc:260] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:41.669484 29254 raft_consensus.cc:3032] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:41.670886 25981 leader_election.cc:302] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:41.611115 26039 rpcz_store.cc:276] 0807 11:10:38.456021 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:38.456277 (+ 256us) service_pool.cc:227] Handling call
0807 11:10:41.567268 (+3110991us) spinlock_profiling.cc:243] Waited 41.4 ms on lock 0x7b5800270998. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc461a2b0 00007f3fc46057f9 00007f3fc46052a2 00007f3fbdceb684
0807 11:10:41.567475 (+ 207us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:41.567661 (+ 186us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:41.567685 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:41.567723 (+ 38us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:41.567754 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:41.567808 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:41.567908 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":95115008}
I20260807 11:10:41.656522 29228 consensus_queue.cc:1035] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260807 11:10:41.589681 25752 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:41.501041 29278 raft_consensus.cc:3032] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:41.609853 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:41.676800 25926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 233) took 1643 ms. Trace:
I20260807 11:10:41.595098 26938 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:41.708719 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:41.708882 25796 raft_consensus.cc:2454] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:10:41.665501 29253 raft_consensus.cc:3032] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:41.711198 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:41.735970 25919 raft_consensus.cc:2379] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 5: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
W20260807 11:10:41.731981 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 142409us
I20260807 11:10:41.713336 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:41.716614 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:41.720667 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:41.737120 25752 leader_election.cc:302] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:41.737808 26038 raft_consensus.cc:2396] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 11:10:41.672065 26046 raft_consensus.cc:2396] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 11:10:41.606773 28244 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:41.718138 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:41.726794 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:41.729533 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:41.741258 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:41.741766 25803 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:41.710130 29273 raft_consensus.cc:2781] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:41.713022 29342 raft_consensus.cc:729] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:41.746464 24034 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260807 11:10:41.706526 29118 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:41.726739 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 55926.5us
I20260807 11:10:41.728722 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:41.727068 25750 leader_election.cc:302] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:41.738688 25752 leader_election.cc:302] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:41.732879 25916 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:41.742156 26985 raft_consensus.cc:2454] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 5.
W20260807 11:10:41.743032 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 148005us
I20260807 11:10:41.733474 25921 raft_consensus.cc:3037] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:41.741068 26986 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 1.
I20260807 11:10:41.724460 29254 raft_consensus.cc:729] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:41.616215 29352 consensus_queue.cc:237] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:41.730756 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:41.733032 25926 rpcz_store.cc:276] 0807 11:10:40.033823 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:40.054554 (+ 20731us) service_pool.cc:227] Handling call
0807 11:10:40.054992 (+ 438us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:40.111345 (+ 56353us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:40.111382 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:41.392184 (+1280802us) spinlock_profiling.cc:243] Waited 2.56 ms on lock 0x7b1800048880. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbc769f96 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbe0a7e11 00007f3fbe0b7442 00007f3fbf6ffc11 00007f3fbf6c458a
0807 11:10:41.392613 (+ 429us) spinlock_profiling.cc:243] Waited 1.52 ms on lock 0x7b2c0004b3f8. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbe0a7e11 00007f3fbe0b7442 00007f3fbf6ffc11 00007f3fbf6c458a 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772
0807 11:10:41.393188 (+ 575us) log.cc:842] Serialized 26 byte log entry
0807 11:10:41.510378 (+117190us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:41.510423 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:41.512682 (+ 2259us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:41.676470 (+163788us) raft_consensus.cc:1671] finished
0807 11:10:41.676500 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:41.676698 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8604,"raft.queue_time_us":33621,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":13296,"spinlock_wait_cycles":9379968,"thread_start_us":158315,"threads_started":2,"wal-append.queue_time_us":239797}
W20260807 11:10:41.943882 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 205250us
I20260807 11:10:41.943161 25932 raft_consensus.cc:3037] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:41.881127 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:41.944634 25752 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:41.945317 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:41.903973 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:41.884493 25918 raft_consensus.cc:2454] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 5.
I20260807 11:10:41.812989 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:41.875877 26936 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:41.730513 29386 raft_consensus.cc:482] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:41.949433 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 222437us
I20260807 11:10:41.812286 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:41.949148 26046 raft_consensus.cc:2379] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 5: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:41.950541 25981 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:41.735584 29278 raft_consensus.cc:729] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
W20260807 11:10:42.022625 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 71504.1us
I20260807 11:10:42.022820 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:41.948720 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 72971.4us
I20260807 11:10:42.024135 25750 leader_election.cc:302] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
W20260807 11:10:41.928678 29389 log.cc:924] Time spent T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.202s user 0.001s sys 0.000s
I20260807 11:10:42.046240 26046 raft_consensus.cc:1226] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260807 11:10:42.022564 25752 leader_election.cc:302] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:42.048755 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:42.049803 26039 raft_consensus.cc:2379] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 4: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:42.048208 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:42.051542 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:41.742448 29253 raft_consensus.cc:729] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:41.742733 29127 raft_consensus.cc:2726] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:41.752322 29206 raft_consensus.cc:2781] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:41.752203 29193 raft_consensus.cc:2781] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:10:42.022768 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:42.070070 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:42.077121 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 54655.6us
I20260807 11:10:41.951332 29330 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:42.114035 25752 leader_election.cc:302] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:42.093206 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:10:42.114998 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 56871.3us
I20260807 11:10:42.116025 26936 leader_election.cc:302] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:41.905369 28946 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:42.081801 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:41.738590 28964 consensus_queue.cc:1035] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.306s
I20260807 11:10:42.109655 26978 raft_consensus.cc:2354] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 6. Current term is 7.
I20260807 11:10:42.080145 25921 raft_consensus.cc:2454] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
W20260807 11:10:42.073532 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 123065us
I20260807 11:10:42.052398 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:42.085932 25932 raft_consensus.cc:2454] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:41.793104 29373 raft_consensus.cc:3037] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:41.882005 29315 raft_consensus.cc:2781] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:41.746562 29273 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:41.951145 29397 raft_consensus.cc:2781] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:42.118294 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:42.121352 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:42.135711 29173 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:42.123257 25919 raft_consensus.cc:2379] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 5: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:41.948534 29001 raft_consensus.cc:482] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:42.026053 29321 raft_consensus.cc:2781] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:41.882412 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:42.041687 29340 raft_consensus.cc:482] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:42.235858 29340 raft_consensus.cc:504] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:42.237316 26980 raft_consensus.cc:1261] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260807 11:10:42.237978 29340 leader_election.cc:288] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:10:42.238727 25920 consensus_meta.cc:287] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.539s user 0.009s sys 0.001s
I20260807 11:10:42.131388 25981 leader_election.cc:302] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:10:42.240054 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 110121us
I20260807 11:10:42.241724 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:42.241954 25981 leader_election.cc:302] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:42.242623 25919 raft_consensus.cc:2454] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
W20260807 11:10:42.243049 29123 consensus_meta.cc:287] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.608s user 0.005s sys 0.004s
I20260807 11:10:42.104640 29196 raft_consensus.cc:2781] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:42.243749 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:42.104450 29336 raft_consensus.cc:2781] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:10:42.181440 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:42.074070 29223 raft_consensus.cc:2781] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:42.182229 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:42.182564 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:42.180940 28946 raft_consensus.cc:2726] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:42.289924 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:42.290529 29261 leader_election.cc:288] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:42.112450 29193 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:42.291502 29193 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:42.092119 29142 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:42.188030 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 72117.9us
I20260807 11:10:42.222896 29273 raft_consensus.cc:3037] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:42.199013 26036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1617) took 1162 ms. Trace:
I20260807 11:10:42.296603 26036 rpcz_store.cc:276] 0807 11:10:41.036741 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:41.038101 (+ 1360us) service_pool.cc:227] Handling call
0807 11:10:41.038480 (+ 379us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:41.562470 (+523990us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:42.198671 (+636201us) spinlock_profiling.cc:243] Waited 17.5 ms on lock 0x7b540029e854. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:42.198930 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40331648}
I20260807 11:10:42.298202 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:42.223124 26046 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:42.026170 29303 raft_consensus.cc:2781] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:41.906209 29118 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:42.138543 29179 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:42.206740 29169 raft_consensus.cc:3032] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:42.227383 29397 raft_consensus.cc:482] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:41.942078 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:42.180140 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:10:42.283910 25803 consensus_meta.cc:287] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.542s user 0.009s sys 0.001s
I20260807 11:10:42.244133 25981 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:42.386875 25803 raft_consensus.cc:2454] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 7.
I20260807 11:10:42.290611 29340 raft_consensus.cc:2781] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:42.291363 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:42.390053 25916 raft_consensus.cc:2454] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:10:42.290638 25926 raft_consensus.cc:2379] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 4: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:42.051561 29305 consensus_queue.cc:1046] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.088s
I20260807 11:10:42.109280 29393 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:42.191814 26039 raft_consensus.cc:1261] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 11:10:42.291446 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:42.293294 28244 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:42.195550 29127 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:42.244549 25919 raft_consensus.cc:3037] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:42.395126 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:42.306990 29303 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:42.301148 29336 raft_consensus.cc:686] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 6 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
W20260807 11:10:42.301648 29399 log.cc:924] Time spent T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.118s user 0.000s sys 0.000s
I20260807 11:10:42.301846 29261 consensus_queue.cc:1035] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:10:42.127066 29330 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:42.052415 29342 consensus_queue.cc:260] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:42.313597 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 199650us
I20260807 11:10:42.407950 25752 leader_election.cc:302] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:42.239138 25920 raft_consensus.cc:2454] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:42.409348 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:42.410308 25920 raft_consensus.cc:2379] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 5: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:42.243558 29196 raft_consensus.cc:482] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:42.222743 29013 raft_consensus.cc:2726] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:42.299018 26036 raft_consensus.cc:2454] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:10:42.382540 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:42.289836 29223 raft_consensus.cc:482] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:42.394474 29179 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:42.233601 29321 raft_consensus.cc:482] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 11:10:42.388665 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 144997us
I20260807 11:10:42.306982 29142 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:42.421859 29179 raft_consensus.cc:504] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:42.397879 29295 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:42.423465 29142 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:42.293035 29029 raft_consensus.cc:2726] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:42.126820 29254 consensus_queue.cc:260] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:42.435374 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:42.436082 29254 raft_consensus.cc:3037] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:42.436705 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:42.437280 29330 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:42.438921 29118 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:42.296005 29244 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:42.396024 26042 raft_consensus.cc:2396] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 11:10:42.439617 29342 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:42.300107 25872 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:10:42.441015 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 140989us
I20260807 11:10:42.447252 29336 consensus_queue.cc:237] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:42.380543 29306 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:42.449076 29306 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:42.450734 25918 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:42.403194 29303 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:42.451808 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1705) took 2903 ms. Trace:
I20260807 11:10:42.314146 29169 raft_consensus.cc:729] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:42.453781 29169 consensus_queue.cc:260] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:42.454816 29388 log.cc:924] Time spent T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.358s user 0.002s sys 0.000s
I20260807 11:10:42.455574 29169 raft_consensus.cc:3037] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:42.456236 29315 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:42.456727 29315 raft_consensus.cc:3037] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:42.402184 29127 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:42.452206 26037 rpcz_store.cc:276] 0807 11:10:39.548513 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:39.548854 (+ 341us) service_pool.cc:227] Handling call
0807 11:10:39.587325 (+ 38471us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:39.896816 (+309491us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:42.451546 (+2554730us) spinlock_profiling.cc:243] Waited 309 ms on lock 0x7b540029de54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:42.451745 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":710292736}
I20260807 11:10:42.458811 29127 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:42.458860 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:42.459678 26037 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:42.399534 29393 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:42.460487 25921 raft_consensus.cc:3037] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:42.461568 29393 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:42.462198 29393 consensus_queue.cc:1035] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:10:42.433673 29321 raft_consensus.cc:3037] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:42.244860 29001 raft_consensus.cc:504] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:42.467758 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:42.468694 29001 leader_election.cc:288] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:42.470470 29244 consensus_queue.cc:1046] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:10:42.471940 29244 consensus_queue.cc:1046] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:10:42.382465 29397 raft_consensus.cc:3037] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:42.052484 29386 raft_consensus.cc:504] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:42.416487 26046 raft_consensus.cc:2454] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:42.475457 29386 leader_election.cc:288] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:42.416539 26938 leader_election.cc:302] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:10:42.384047 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 78617us
W20260807 11:10:42.476572 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 60125.9us
I20260807 11:10:42.476732 29244 raft_consensus.cc:3032] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:42.477187 29244 raft_consensus.cc:729] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:42.393687 29340 raft_consensus.cc:482] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:42.478113 29334 raft_consensus.cc:3032] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:42.478420 29340 raft_consensus.cc:3037] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:42.478643 29334 raft_consensus.cc:729] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:42.478876 29380 raft_consensus.cc:2781] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:42.479333 29380 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:42.479698 29380 raft_consensus.cc:3037] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:42.479678 29334 consensus_queue.cc:260] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:42.420073 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:42.478168 26936 leader_election.cc:302] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:42.424209 29307 consensus_queue.cc:1046] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 11:10:42.425300 29047 consensus_queue.cc:1035] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:10:42.425853 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:42.426723 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:42.427788 26987 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 407) took 2960 ms. Trace:
I20260807 11:10:42.428196 28964 raft_consensus.cc:2781] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:42.428648 28438 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:42.414455 29196 raft_consensus.cc:3037] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:42.429394 29261 raft_consensus.cc:482] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:42.509420 26936 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:42.430897 29206 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:42.433110 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:42.433878 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:42.437227 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:42.451252 29306 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:42.457480 29295 raft_consensus.cc:2726] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:42.400252 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:42.434144 29347 consensus_meta.cc:287] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.885s user 0.002s sys 0.010s
I20260807 11:10:42.470791 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:42.217573 29253 consensus_queue.cc:260] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:42.529096 29261 raft_consensus.cc:504] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:42.290472 26976 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 412) took 1312 ms. Trace:
I20260807 11:10:42.314329 29370 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:42.568941 29370 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:42.569607 29193 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:42.570776 29261 leader_election.cc:288] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:42.533471 29184 consensus_meta.cc:287] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.915s user 0.010s sys 0.000s
I20260807 11:10:42.571784 29193 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:42.571950 29370 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:42.397347 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:42.561486 29334 raft_consensus.cc:3037] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:42.490377 25872 leader_election.cc:302] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:42.527937 28438 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:42.419800 25919 raft_consensus.cc:2454] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 5.
I20260807 11:10:42.493325 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:10:42.493527 25981 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:10:42.500376 25869 connection.cc:232] RPC call timeout handler was delayed by -2.4347504370380193s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15180000000000002s for a response
I20260807 11:10:42.498122 26987 rpcz_store.cc:276] 0807 11:10:39.467188 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:39.483686 (+ 16498us) service_pool.cc:227] Handling call
0807 11:10:40.444426 (+960740us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:40.491352 (+ 46926us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:42.427579 (+1936227us) spinlock_profiling.cc:243] Waited 906 ms on lock 0x7b54006b26d0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:42.427723 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2083076992}
I20260807 11:10:42.503934 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:42.512105 29228 raft_consensus.cc:2781] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 11:10:42.417582 29223 raft_consensus.cc:3037] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:42.515650 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:42.517118 25798 raft_consensus.cc:2396] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260807 11:10:42.521281 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:42.530505 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:42.531028 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:42.545209 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1620) took 3019 ms. Trace:
I20260807 11:10:42.501787 28964 raft_consensus.cc:482] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:42.611503 28964 raft_consensus.cc:3037] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:42.547654 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:42.528857 29092 raft_consensus.cc:2726] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:42.528622 29206 raft_consensus.cc:3037] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:42.550736 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:42.554888 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:42.615603 26985 raft_consensus.cc:2454] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:42.616925 29306 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:42.617478 29306 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:42.620352 29306 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:42.556932 29347 raft_consensus.cc:2726] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195) with higher term. Message: Invalid argument: T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5.
W20260807 11:10:42.525112 26936 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.120.130:36465 timed out after 1.848s (SENT) [suppressed 13 similar messages]
I20260807 11:10:42.494567 29173 raft_consensus.cc:2781] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:10:42.492218 29118 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:42.497228 29029 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:42.566992 29047 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:42.572782 25916 raft_consensus.cc:2454] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
W20260807 11:10:42.573787 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 83471.4us
W20260807 11:10:42.626590 29231 consensus_meta.cc:287] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.989s user 0.010s sys 0.001s
W20260807 11:10:42.575537 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 82084.9us
I20260807 11:10:42.645846 25981 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:42.647857 25981 leader_election.cc:302] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:42.574237 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:42.501314 29308 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:42.586714 25869 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
W20260807 11:10:42.654788 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 68196.6us
I20260807 11:10:42.498343 29244 consensus_queue.cc:260] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:42.591709 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:42.592811 26987 raft_consensus.cc:1261] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260807 11:10:42.593286 28438 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:42.579741 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:10:42.572706 29198 consensus_meta.cc:287] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.929s user 0.007s sys 0.004s
I20260807 11:10:42.601193 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:42.590688 29261 raft_consensus.cc:2781] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:42.551654 29303 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:42.605717 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:42.610214 26986 raft_consensus.cc:1261] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260807 11:10:42.623960 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:42.507083 29386 consensus_queue.cc:1046] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.087s
I20260807 11:10:42.565722 26976 rpcz_store.cc:276] 0807 11:10:40.977915 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:40.978790 (+ 875us) service_pool.cc:227] Handling call
0807 11:10:42.290274 (+1311484us) spinlock_profiling.cc:243] Waited 720 ms on lock 0x7b54004718d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:42.290405 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1657147904}
I20260807 11:10:42.581279 29184 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:42.661073 29330 raft_consensus.cc:2781] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:10:42.574581 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:42.623085 29370 raft_consensus.cc:2781] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election won for term 5
W20260807 11:10:42.651024 29310 consensus_meta.cc:287] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.938s user 0.008s sys 0.001s
W20260807 11:10:42.652256 29401 log.cc:924] Time spent T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.146s user 0.001s sys 0.000s
I20260807 11:10:42.722525 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:42.644989 29029 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:42.655279 25981 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:42.575059 29283 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:42.548382 29321 raft_consensus.cc:504] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:42.712941 29244 raft_consensus.cc:3037] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:42.629535 29173 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:42.629856 29228 raft_consensus.cc:686] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:42.661628 25869 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:10:42.766078 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 104565us
I20260807 11:10:42.766858 25869 leader_election.cc:302] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:42.662828 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:42.666317 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:42.676349 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:42.768082 25796 raft_consensus.cc:2379] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 5: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:42.660342 29308 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:42.735108 29370 raft_consensus.cc:686] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 5 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:42.655875 29347 raft_consensus.cc:3037] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:42.660591 29295 raft_consensus.cc:2781] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:42.709802 25919 raft_consensus.cc:3037] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:42.718639 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:42.719749 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:42.778946 25799 raft_consensus.cc:2454] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:42.641721 29118 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:42.721519 25920 raft_consensus.cc:2454] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:42.590045 25925 rpcz_store.cc:276] 0807 11:10:39.497754 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:39.577828 (+ 80074us) service_pool.cc:227] Handling call
0807 11:10:41.335031 (+1757203us) spinlock_profiling.cc:243] Waited 2.35 ms on lock 0x7b58002e0e98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc461a2b0 00007f3fc46057f9 00007f3fc46052a2 00007f3fbdceb684
0807 11:10:41.335305 (+ 274us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:10:41.379281 (+ 43976us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:41.379323 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 11:10:41.380074 (+ 751us) log.cc:842] Serialized 52 byte log entry
0807 11:10:41.705206 (+325132us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:10:41.709207 (+ 4001us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:41.716509 (+ 7302us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:42.454216 (+737707us) spinlock_profiling.cc:243] Waited 5.75 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:10:42.491399 (+ 37183us) raft_consensus.cc:1671] finished
0807 11:10:42.491419 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:42.516610 (+ 25191us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2435,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":5125,"spinlock_wait_cycles":18626176,"thread_start_us":291712,"threads_started":1,"wal-append.queue_time_us":524911}
I20260807 11:10:42.725613 25798 raft_consensus.cc:2454] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:42.785459 25806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 292) took 1122 ms. Trace:
I20260807 11:10:42.654063 29047 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:42.704640 29006 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:42.672717 29261 raft_consensus.cc:482] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
W20260807 11:10:42.747114 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 91986.3us
I20260807 11:10:42.799700 25981 leader_election.cc:302] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:42.754459 25750 leader_election.cc:398] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7.
I20260807 11:10:42.659402 26035 raft_consensus.cc:2454] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:42.800436 25750 leader_election.cc:401] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
W20260807 11:10:42.801330 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 47055.5us
I20260807 11:10:42.762595 25932 raft_consensus.cc:2396] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260807 11:10:42.753084 26975 raft_consensus.cc:2379] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 7: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:42.770985 25797 raft_consensus.cc:2354] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3.
I20260807 11:10:42.499287 25921 raft_consensus.cc:2454] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
W20260807 11:10:42.768873 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 277412us
I20260807 11:10:42.733865 29330 raft_consensus.cc:686] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:42.750739 29266 raft_consensus.cc:2726] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 11:10:42.776264 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:42.721401 29184 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:42.769452 26974 raft_consensus.cc:2354] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 5. Current term is 6.
I20260807 11:10:42.765307 29173 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:42.787477 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:42.822564 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:42.772284 29001 raft_consensus.cc:2781] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:42.775748 29169 raft_consensus.cc:2726] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:42.758049 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:42.824222 26037 raft_consensus.cc:2454] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:42.800450 25919 raft_consensus.cc:2454] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 5.
I20260807 11:10:42.785862 25806 rpcz_store.cc:276] 0807 11:10:41.662739 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:41.665250 (+ 2511us) service_pool.cc:227] Handling call
0807 11:10:41.665635 (+ 385us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:41.666011 (+ 376us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:41.666042 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:41.666520 (+ 478us) log.cc:842] Serialized 26 byte log entry
0807 11:10:41.845508 (+178988us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:41.845567 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:41.845657 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:42.785247 (+939590us) raft_consensus.cc:1671] finished
0807 11:10:42.785268 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:42.785400 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":266,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":83,"thread_start_us":106078,"threads_started":1,"wal-append.queue_time_us":735044}
I20260807 11:10:42.806234 25981 leader_election.cc:302] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:42.825253 25750 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:42.826627 25750 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:42.826965 25981 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
W20260807 11:10:42.786862 29373 consensus_meta.cc:287] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.561s user 0.006s sys 0.005s
I20260807 11:10:42.809297 25796 raft_consensus.cc:2454] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:10:42.786682 29295 raft_consensus.cc:686] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:42.783594 29386 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:42.816216 25808 raft_consensus.cc:2454] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
W20260807 11:10:42.820320 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 295219us
I20260807 11:10:42.836469 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:42.828500 25750 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:42.788862 29358 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:42.838531 26986 raft_consensus.cc:1261] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260807 11:10:42.806942 28946 raft_consensus.cc:2726] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7.
I20260807 11:10:42.783984 29127 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:42.752457 29283 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:42.834364 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:42.807829 29261 raft_consensus.cc:3037] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:42.835314 26034 raft_consensus.cc:2396] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260807 11:10:42.837105 25799 raft_consensus.cc:2454] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:42.830299 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:42.842353 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:42.855469 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:42.856915 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:42.857961 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:42.845948 29127 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:42.859927 29047 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:42.860245 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:42.860437 29127 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:42.860545 29047 raft_consensus.cc:2781] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:42.861024 29047 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:42.861055 29127 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:42.861517 29047 raft_consensus.cc:3037] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:42.835718 29266 raft_consensus.cc:2781] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:42.862406 29266 raft_consensus.cc:686] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:42.863962 26037 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
W20260807 11:10:42.830017 29318 consensus_meta.cc:287] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 1.302s user 0.007s sys 0.005s
I20260807 11:10:42.850708 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:42.865510 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:42.840534 26936 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:42.829560 29001 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 11:10:42.851845 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 64556.7us
I20260807 11:10:42.861435 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:42.854161 25796 raft_consensus.cc:2454] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:42.835873 29352 raft_consensus.cc:2781] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election won for term 5
W20260807 11:10:42.846840 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 67447.9us
I20260807 11:10:42.867749 26936 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:42.835152 29254 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:42.881363 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 52938.7us
I20260807 11:10:42.830905 29334 consensus_queue.cc:1035] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260807 11:10:42.872736 25931 raft_consensus.cc:1261] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260807 11:10:42.874843 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:42.807690 29006 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:42.847572 29411 raft_consensus.cc:482] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:42.878223 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:42.882246 25750 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:42.842039 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:42.799768 29380 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:42.840335 29386 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:42.871515 25869 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:42.873931 29001 raft_consensus.cc:3037] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:42.885224 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:42.935984 25799 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:42.850971 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:42.876638 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:42.877578 29231 raft_consensus.cc:2781] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:42.887920 29352 raft_consensus.cc:686] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 5 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:42.861639 29127 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:42.954782 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:42.893409 29295 consensus_queue.cc:237] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:42.894771 26975 raft_consensus.cc:1261] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
W20260807 11:10:42.895203 29409 log.cc:924] Time spent T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.139s user 0.001s sys 0.000s
I20260807 11:10:42.895385 29029 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:42.869954 29336 raft_consensus.cc:2726] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:42.866207 29266 consensus_queue.cc:237] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:42.981326 29352 consensus_queue.cc:237] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:42.982754 29340 consensus_meta.cc:287] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.504s user 0.007s sys 0.002s
I20260807 11:10:42.982998 29047 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:42.899746 25752 leader_election.cc:302] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:42.990265 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 90584.6us
I20260807 11:10:42.885581 29254 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:42.883389 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:42.994105 26036 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:10:42.995225 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:42.995879 26036 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:42.880553 29013 raft_consensus.cc:482] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:42.932472 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 100191us
I20260807 11:10:43.001224 29205 raft_consensus.cc:3032] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:43.002070 29205 raft_consensus.cc:729] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:42.936218 26936 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:10:43.003999 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 67917.4us
I20260807 11:10:43.003724 29205 consensus_queue.cc:260] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:43.007485 29417 raft_consensus.cc:2781] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:10:42.941265 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:42.945071 26037 raft_consensus.cc:2454] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:43.012869 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:42.940200 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:42.837702 29358 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:42.949733 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:43.017393 25799 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:43.018003 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:43.018671 25796 raft_consensus.cc:2454] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:43.019670 29358 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:42.869426 29318 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:42.879473 29270 raft_consensus.cc:2726] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:42.839076 29228 consensus_queue.cc:237] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:42.959848 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 88457.4us
I20260807 11:10:43.001217 29013 raft_consensus.cc:504] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:43.022109 25869 leader_election.cc:302] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:43.022419 29318 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:42.947014 29231 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:43.023314 29231 raft_consensus.cc:3037] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:43.023465 25869 leader_election.cc:302] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:43.024120 29013 leader_election.cc:288] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:42.885210 29415 raft_consensus.cc:2726] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:10:42.901751 28244 raft_consensus.cc:2781] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:43.025045 25801 raft_consensus.cc:2354] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6.
I20260807 11:10:42.863984 29370 consensus_queue.cc:237] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:42.932153 25925 raft_consensus.cc:3037] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:42.941861 29303 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:42.983290 29340 raft_consensus.cc:504] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:43.029759 26039 raft_consensus.cc:1226] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260807 11:10:42.867277 29418 raft_consensus.cc:2781] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:10:42.991055 25752 leader_election.cc:302] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:42.870936 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:43.008023 29417 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:43.019857 25796 raft_consensus.cc:1226] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:43.025326 28244 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:42.934408 29397 raft_consensus.cc:504] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:43.045228 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 54242.5us
I20260807 11:10:42.971594 29431 raft_consensus.cc:2726] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:43.045233 26036 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 7.
I20260807 11:10:42.963917 29308 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:42.965031 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:42.973750 29321 leader_election.cc:288] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:42.992017 29334 raft_consensus.cc:2726] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:43.033798 29340 leader_election.cc:288] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:43.068984 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:43.061508 25919 raft_consensus.cc:2454] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:10:42.946856 29407 consensus_queue.cc:1035] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.085s
I20260807 11:10:43.042088 29118 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:43.076074 26036 raft_consensus.cc:2396] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260807 11:10:43.079049 26036 raft_consensus.cc:1226] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:43.059705 29318 raft_consensus.cc:2781] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:43.057217 29417 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:43.061384 28244 raft_consensus.cc:3037] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:42.901204 29411 raft_consensus.cc:504] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:42.900900 29330 consensus_queue.cc:237] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:43.087837 26938 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:43.061820 29418 raft_consensus.cc:686] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:43.063907 29184 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:43.087764 26036 raft_consensus.cc:1261] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260807 11:10:43.061272 26979 raft_consensus.cc:2454] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:43.105852 25925 raft_consensus.cc:2454] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
W20260807 11:10:43.092886 29315 consensus_meta.cc:287] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.631s user 0.005s sys 0.006s
I20260807 11:10:43.115758 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:43.120025 25925 raft_consensus.cc:2379] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 5: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:43.083312 29431 raft_consensus.cc:2726] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 11:10:43.068312 28761 consensus_queue.cc:1035] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260807 11:10:43.084272 29318 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:43.121991 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:43.082408 29438 raft_consensus.cc:2726] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:43.134636 25981 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:43.142186 25750 leader_election.cc:302] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:43.142498 25981 leader_election.cc:302] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:43.140909 26036 raft_consensus.cc:2454] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
W20260807 11:10:43.209544 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 67156.9us
I20260807 11:10:43.209661 26938 leader_election.cc:302] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:43.210430 25981 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:43.117020 29207 raft_consensus.cc:2781] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 11:10:43.214859 25981 leader_election.cc:302] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:43.216473 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:43.217454 25925 raft_consensus.cc:2454] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:10:43.218586 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:43.219159 25925 raft_consensus.cc:3037] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:43.206220 29370 raft_consensus.cc:2726] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 11:10:43.224972 29433 log.cc:924] Time spent T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.250s user 0.001s sys 0.000s
I20260807 11:10:43.225356 29303 raft_consensus.cc:2781] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260807 11:10:43.226389 29423 log.cc:924] Time spent T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.295s user 0.000s sys 0.001s
W20260807 11:10:43.229398 29432 log.cc:924] Time spent T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.134s user 0.001s sys 0.000s
I20260807 11:10:43.229610 29358 raft_consensus.cc:2781] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:43.225169 29307 raft_consensus.cc:2781] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:10:43.233471 29370 raft_consensus.cc:2726] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:43.147825 29318 raft_consensus.cc:3037] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:43.216029 26036 raft_consensus.cc:1261] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260807 11:10:43.295002 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:10:43.153890 29273 consensus_meta.cc:287] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.839s user 0.010s sys 0.001s
I20260807 11:10:43.299482 29273 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:43.212212 29334 raft_consensus.cc:2781] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:43.281041 25925 raft_consensus.cc:2454] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:10:43.282997 25750 leader_election.cc:302] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:43.331975 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:43.333108 25925 raft_consensus.cc:2454] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
W20260807 11:10:43.337706 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 54824.1us
I20260807 11:10:43.275640 29303 raft_consensus.cc:482] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:43.341355 25750 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:43.273916 29207 raft_consensus.cc:686] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:43.280376 29117 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:43.286290 29380 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:43.280211 29386 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:43.365897 29047 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:43.370414 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 155643us
I20260807 11:10:43.370659 25799 raft_consensus.cc:2454] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:10:43.385836 25981 leader_election.cc:398] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 election: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 6. Current term is 7.
I20260807 11:10:43.386656 26936 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:43.371163 25752 leader_election.cc:302] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:43.373275 28761 raft_consensus.cc:2781] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:10:43.356716 28946 raft_consensus.cc:3037] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:43.286779 29358 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:43.376065 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:43.377725 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 354317us
I20260807 11:10:43.293473 29307 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:43.122635 29315 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:43.400600 29127 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:43.401600 25981 leader_election.cc:302] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:43.429931 25981 leader_election.cc:302] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:43.373579 29414 log.cc:924] Time spent T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.233s user 0.000s sys 0.000s
I20260807 11:10:43.302042 29273 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:43.351709 29334 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:43.337654 29321 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:43.413188 25752 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:43.377050 29425 log.cc:924] Time spent T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.389s user 0.002s sys 0.000s
I20260807 11:10:43.414886 25869 leader_election.cc:302] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:43.366142 29303 raft_consensus.cc:3037] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:43.423244 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:43.426045 29397 leader_election.cc:288] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:43.426895 29207 consensus_queue.cc:237] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:43.374518 29250 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:43.434249 25981 leader_election.cc:302] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:43.374506 29254 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:43.403596 29380 raft_consensus.cc:482] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:43.508713 28449 raft_consensus.cc:3032] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 LEADER]: Stepping down as leader of term 6
W20260807 11:10:43.429988 28964 consensus_meta.cc:287] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.818s user 0.011s sys 0.002s
I20260807 11:10:43.509155 29380 raft_consensus.cc:504] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:43.509708 28964 raft_consensus.cc:504] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:43.511364 28244 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:43.445672 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:43.509189 28449 raft_consensus.cc:729] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:43.516729 25925 raft_consensus.cc:2454] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:43.518113 25750 leader_election.cc:302] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:43.446798 25869 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:43.519954 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 73239.7us
I20260807 11:10:43.447647 25752 leader_election.cc:302] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:43.447825 29373 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:43.434129 29117 raft_consensus.cc:2726] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:43.430474 28761 raft_consensus.cc:686] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:43.448923 29445 raft_consensus.cc:2726] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260807 11:10:43.453222 29223 consensus_meta.cc:287] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.847s user 0.004s sys 0.009s
I20260807 11:10:43.455476 25800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:53306 (request call id 702) took 1916 ms. Trace:
I20260807 11:10:43.456236 29411 leader_election.cc:288] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:43.457892 29315 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:43.553079 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:10:43.459688 25921 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1642) took 1834 ms (client timeout 2095 ms). Trace:
I20260807 11:10:43.460690 25920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1636) took 1843 ms. Trace:
I20260807 11:10:43.554754 25920 rpcz_store.cc:276] 0807 11:10:41.617501 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:42.419458 (+801957us) service_pool.cc:227] Handling call
0807 11:10:43.460513 (+1041055us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b5400362ed4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:43.460627 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":334438784}
W20260807 11:10:43.554646 25921 rpcz_store.cc:269] 0807 11:10:41.624972 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:42.178931 (+553959us) service_pool.cc:227] Handling call
0807 11:10:43.459483 (+1280552us) spinlock_profiling.cc:243] Waited 73.6 ms on lock 0x7b5400052b54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:43.459624 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":169244032}
I20260807 11:10:43.556135 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:43.556305 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:43.556886 25920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1639) took 1932 ms. Trace:
I20260807 11:10:43.557192 25921 raft_consensus.cc:2454] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:43.465663 29127 raft_consensus.cc:2726] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:43.463469 25796 raft_consensus.cc:1226] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:43.431447 29358 raft_consensus.cc:3037] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:43.558634 28244 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:43.558957 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:43.559756 25796 raft_consensus.cc:2379] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 5: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:43.430882 29307 raft_consensus.cc:3037] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
W20260807 11:10:43.459741 29206 consensus_meta.cc:287] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.846s user 0.007s sys 0.006s
I20260807 11:10:43.561264 29206 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:43.562412 26046 raft_consensus.cc:1226] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260807 11:10:43.432796 28911 raft_consensus.cc:2726] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:43.440203 29439 raft_consensus.cc:2726] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:43.445137 29334 raft_consensus.cc:3037] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:43.564039 29206 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:10:43.503252 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 69081.9us
W20260807 11:10:43.507179 29001 consensus_meta.cc:287] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.558s user 0.010s sys 0.001s
I20260807 11:10:43.430876 29270 raft_consensus.cc:2781] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:10:43.564704 29001 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:43.467281 29261 consensus_meta.cc:287] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.595s user 0.009s sys 0.003s
I20260807 11:10:43.565709 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:43.566037 29261 raft_consensus.cc:504] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:43.566771 29001 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:43.511780 29380 leader_election.cc:288] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:43.516201 29173 consensus_meta.cc:287] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.680s user 0.008s sys 0.003s
I20260807 11:10:43.442763 25799 raft_consensus.cc:1226] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:43.567477 29173 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:43.568156 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:43.515080 29231 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:43.568748 25799 raft_consensus.cc:1261] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20260807 11:10:43.569142 25799 pending_rounds.cc:76] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815: Aborting all ops after (but not including) 3
I20260807 11:10:43.569361 25799 pending_rounds.cc:97] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260807 11:10:43.569444 29173 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:43.569600 25799 raft_consensus.cc:2866] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:10:43.520649 25869 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:43.570113 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:43.570348 29231 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:43.570767 26975 raft_consensus.cc:3037] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
W20260807 11:10:43.524296 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 76728.5us
I20260807 11:10:43.571473 26987 raft_consensus.cc:1226] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260807 11:10:43.554339 25925 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:43.556490 25750 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:43.557286 25920 rpcz_store.cc:276] 0807 11:10:41.624081 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:43.555853 (+1931772us) service_pool.cc:227] Handling call
0807 11:10:43.556834 (+ 981us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:43.565074 29270 raft_consensus.cc:686] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:43.573556 29270 consensus_queue.cc:237] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:43.572376 29261 leader_election.cc:288] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:43.570390 25926 raft_consensus.cc:2379] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 7: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
W20260807 11:10:43.499420 29244 consensus_meta.cc:287] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.732s user 0.006s sys 0.006s
I20260807 11:10:43.464339 29318 raft_consensus.cc:504] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:43.579250 25806 raft_consensus.cc:2454] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:43.579421 25925 raft_consensus.cc:2454] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:10:43.511930 28964 leader_election.cc:288] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:43.582149 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:43.580493 29334 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:43.582391 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:43.583102 25921 raft_consensus.cc:2454] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:10:43.584283 29318 leader_election.cc:288] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:43.585228 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:43.585788 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:43.586624 25925 raft_consensus.cc:2454] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:43.586946 29334 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:43.582464 26975 raft_consensus.cc:2454] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:43.576712 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:43.594328 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:43.595366 25925 raft_consensus.cc:2379] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 5: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:43.596246 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:43.596858 25925 raft_consensus.cc:2454] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:43.596741 26936 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:43.597553 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1552) took 1080 ms. Trace:
I20260807 11:10:43.597923 25925 rpcz_store.cc:276] 0807 11:10:42.517300 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:43.595970 (+1078670us) service_pool.cc:227] Handling call
0807 11:10:43.597508 (+ 1538us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:43.598770 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:43.599293 25925 raft_consensus.cc:3037] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:43.595517 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:43.600076 25920 raft_consensus.cc:3037] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:43.551312 25800 rpcz_store.cc:276] 0807 11:10:41.539296 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:41.580200 (+ 40904us) service_pool.cc:227] Handling call
0807 11:10:43.454891 (+1874691us) spinlock_profiling.cc:243] Waited 33 ms on lock 0x7b54002f4c54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649af0 00007f3fbf6ccf2a 00007f3fc4609c6e 00007f3fbdd00807 00007f3fbdd00792 00007f3fbdd006c1 00007f3fbdd0063c 00007f3fbdcff8f2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:43.455355 (+ 464us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":75870976}
I20260807 11:10:43.606513 25921 raft_consensus.cc:2454] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:43.608449 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:43.523138 29254 raft_consensus.cc:2726] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:43.610070 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1550) took 1174 ms. Trace:
I20260807 11:10:43.606549 25925 raft_consensus.cc:2454] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:43.608614 28449 consensus_queue.cc:260] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:43.413249 29418 consensus_queue.cc:237] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:43.480108 29321 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:43.523454 29250 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:43.550017 29373 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:43.618789 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:43.754719 25798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1641) took 1188 ms. Trace:
I20260807 11:10:43.751761 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1653) took 1705 ms. Trace:
I20260807 11:10:43.742022 25920 raft_consensus.cc:2454] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:43.634987 25921 rpcz_store.cc:276] 0807 11:10:42.432621 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:43.585332 (+1152711us) service_pool.cc:227] Handling call
0807 11:10:43.607188 (+ 21856us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:43.746304 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:43.803300 25798 rpcz_store.cc:276] 0807 11:10:42.566010 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:42.591382 (+ 25372us) service_pool.cc:227] Handling call
0807 11:10:43.754537 (+1163155us) spinlock_profiling.cc:243] Waited 66 ms on lock 0x7b5400137254. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:43.754657 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":151686528}
W20260807 11:10:43.645103 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 73221us
W20260807 11:10:43.613674 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 93081.7us
I20260807 11:10:43.810639 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:43.811105 25869 leader_election.cc:302] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:43.812793 25869 leader_election.cc:302] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:43.816028 25869 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:43.818624 25869 leader_election.cc:302] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:43.803903 25932 rpcz_store.cc:276] 0807 11:10:42.046510 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:42.495062 (+448552us) service_pool.cc:227] Handling call
0807 11:10:43.751529 (+1256467us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b54001def54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:43.751694 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":418220928}
I20260807 11:10:43.809216 25796 raft_consensus.cc:2454] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:43.821833 25869 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
W20260807 11:10:43.823700 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 267262us
I20260807 11:10:43.824986 25750 leader_election.cc:302] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:43.709391 25919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35276 (request call id 239) took 1097 ms. Trace:
I20260807 11:10:43.826179 25919 rpcz_store.cc:276] 0807 11:10:42.612185 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:42.855003 (+242818us) service_pool.cc:227] Handling call
0807 11:10:43.709140 (+854137us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b54000d1954. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:43.709309 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":402402688}
I20260807 11:10:43.538715 29440 raft_consensus.cc:2781] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260807 11:10:43.725127 29196 consensus_meta.cc:287] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 1.197s user 0.010s sys 0.003s
I20260807 11:10:43.831588 25797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1507) took 1398 ms. Trace:
W20260807 11:10:43.611368 29447 log.cc:924] Time spent T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.093s user 0.001s sys 0.002s
I20260807 11:10:43.555018 29397 raft_consensus.cc:2781] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260807 11:10:43.611425 29414 log.cc:924] Time spent T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
W20260807 11:10:43.738973 29417 consensus_meta.cc:287] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.633s user 0.008s sys 0.000s
I20260807 11:10:43.831630 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:43.871456 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:43.871891 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 46970.2us
I20260807 11:10:43.872363 25799 raft_consensus.cc:2454] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 5.
I20260807 11:10:43.872799 25750 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:43.722627 29334 raft_consensus.cc:2781] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:43.863128 25925 raft_consensus.cc:2454] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:43.864019 25797 rpcz_store.cc:276] 0807 11:10:42.433188 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:42.436524 (+ 3336us) service_pool.cc:227] Handling call
0807 11:10:43.831381 (+1394857us) spinlock_profiling.cc:243] Waited 302 ms on lock 0x7b54002f6554. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:43.831521 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":695523072}
I20260807 11:10:43.622251 29173 raft_consensus.cc:2741] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260807 11:10:43.605696 29223 raft_consensus.cc:504] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:43.722437 29318 consensus_queue.cc:1046] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.163s
I20260807 11:10:43.831688 29330 raft_consensus.cc:2781] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:43.612733 29454 raft_consensus.cc:2781] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:43.878204 25800 raft_consensus.cc:2379] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 4: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:43.834693 29440 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:43.877709 25750 leader_election.cc:302] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:43.831835 29277 raft_consensus.cc:2781] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:43.831992 29001 raft_consensus.cc:2726] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:43.910830 25750 leader_election.cc:302] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:43.746634 28761 consensus_queue.cc:237] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:43.721752 29307 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:43.867133 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:43.828163 29445 raft_consensus.cc:2726] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:43.752390 29358 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:43.879739 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:43.900830 29440 raft_consensus.cc:3037] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:43.916299 25919 raft_consensus.cc:2362] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4. Re-sending same reply.
I20260807 11:10:43.917491 29358 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:43.915711 25750 leader_election.cc:302] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:43.918124 26036 raft_consensus.cc:3037] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:43.918377 29127 raft_consensus.cc:2781] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:10:43.918860 29127 raft_consensus.cc:686] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:43.919737 29127 consensus_queue.cc:237] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:43.870357 29397 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:43.921712 29397 raft_consensus.cc:3037] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:43.878255 29334 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:43.902057 25797 raft_consensus.cc:1226] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:43.925069 29334 raft_consensus.cc:3037] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:43.925400 25797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 302) took 1198 ms. Trace:
I20260807 11:10:43.925758 25797 rpcz_store.cc:276] 0807 11:10:42.727218 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:43.883468 (+1156250us) service_pool.cc:227] Handling call
0807 11:10:43.901932 (+ 18464us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:43.925141 (+ 23209us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:43.925345 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:43.882774 29244 raft_consensus.cc:2781] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:10:43.873634 29196 raft_consensus.cc:504] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:43.903929 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1633) took 1461 ms. Trace:
I20260807 11:10:43.934080 26037 rpcz_store.cc:276] 0807 11:10:42.442437 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:43.014212 (+571775us) service_pool.cc:227] Handling call
0807 11:10:43.014514 (+ 302us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:43.014896 (+ 382us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:43.014940 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:43.015470 (+ 530us) log.cc:842] Serialized 26 byte log entry
0807 11:10:43.116220 (+100750us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:43.116280 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:43.116370 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:43.903732 (+787362us) raft_consensus.cc:1671] finished
0807 11:10:43.903752 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:43.903873 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":126,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":123,"thread_start_us":100368,"threads_started":1,"wal-append.queue_time_us":425374}
I20260807 11:10:43.926733 29315 raft_consensus.cc:2781] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:10:43.935902 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:43.935933 29315 raft_consensus.cc:686] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:43.865120 28244 consensus_queue.cc:1035] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260807 11:10:43.895458 29454 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:43.937304 29315 consensus_queue.cc:237] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:43.938207 29454 raft_consensus.cc:3037] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:43.907294 29373 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:43.880439 28911 consensus_queue.cc:1035] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260807 11:10:43.880580 29417 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:43.662434 28449 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:43.913169 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1615) took 3060 ms. Trace:
I20260807 11:10:43.893276 29330 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:43.917377 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:43.827922 29169 raft_consensus.cc:2726] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260807 11:10:43.913894 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 155229us
I20260807 11:10:43.951593 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:43.952432 25932 raft_consensus.cc:3037] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:43.875556 28293 raft_consensus.cc:2726] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:43.930579 29244 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:43.910578 29277 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:43.940833 26938 leader_election.cc:302] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:43.942888 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 117140us
I20260807 11:10:43.943082 26036 raft_consensus.cc:2454] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:10:43.942544 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:43.963085 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 374) took 1913 ms. Trace:
I20260807 11:10:43.950080 25786 raft_consensus.cc:1226] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:43.927573 29318 raft_consensus.cc:2781] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:10:43.969376 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:43.936555 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1632) took 1494 ms. Trace:
I20260807 11:10:43.970191 25981 leader_election.cc:302] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:43.970238 26037 raft_consensus.cc:3037] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:43.963472 26042 rpcz_store.cc:276] 0807 11:10:42.049420 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:42.440135 (+390715us) service_pool.cc:227] Handling call
0807 11:10:42.440488 (+ 353us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:42.573131 (+132643us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:42.573163 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:42.573211 (+ 48us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:42.573242 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:43.962855 (+1389613us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b5400312254. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:43.962904 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:43.963028 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2499,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":42669,"spinlock_wait_cycles":304227584}
I20260807 11:10:43.926472 29440 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:43.946363 26041 rpcz_store.cc:276] 0807 11:10:40.852835 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:40.964659 (+111824us) service_pool.cc:227] Handling call
0807 11:10:41.008872 (+ 44213us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:42.448596 (+1439724us) spinlock_profiling.cc:243] Waited 1.87 ms on lock 0x7b180035cec0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbf35382c 00007f3fbc77e9dc 00007f3fc55b5a39 00007f3fbc79b8bb 00007f3fbc799cbf 00007f3fbc799155 00007f3fbf6bc3d0 00007f3fbf6bcf6b 00007f3fbf6c778b
0807 11:10:42.448739 (+ 143us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:42.474887 (+ 26148us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:42.475405 (+ 518us) log.cc:842] Serialized 26 byte log entry
0807 11:10:42.734022 (+258617us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:42.867080 (+133058us) log.cc:842] Serialized 14 byte log entry
0807 11:10:42.867288 (+ 208us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:42.867482 (+ 194us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:43.905144 (+1037662us) raft_consensus.cc:1671] finished
0807 11:10:43.912964 (+ 7820us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:43.913094 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":828,"prepare.queue_time_us":215252,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":12007,"raft.queue_time_us":25006,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":14310,"spinlock_wait_cycles":4299008,"thread_start_us":386957,"threads_started":3,"wal-append.queue_time_us":589149}
W20260807 11:10:43.974855 29347 consensus_meta.cc:287] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 1.193s user 0.001s sys 0.011s
I20260807 11:10:43.975307 29347 raft_consensus.cc:2726] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:43.975468 25869 leader_election.cc:302] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:43.976963 29001 consensus_queue.cc:1046] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:10:43.977159 29479 consensus_queue.cc:1035] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:10:43.977838 29001 raft_consensus.cc:2741] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260807 11:10:43.978446 29001 raft_consensus.cc:3032] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:43.978745 29439 raft_consensus.cc:482] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:43.978775 29001 raft_consensus.cc:729] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:43.968909 29207 raft_consensus.cc:2726] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:43.980254 29352 raft_consensus.cc:482] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:43.980747 29352 raft_consensus.cc:504] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:43.982321 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:43.982990 29352 leader_election.cc:288] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:43.983500 29352 raft_consensus.cc:482] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:43.983543 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:43.984047 29352 raft_consensus.cc:504] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:43.985026 29441 raft_consensus.cc:2781] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:10:43.969291 29277 raft_consensus.cc:3037] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:43.991101 25932 raft_consensus.cc:2454] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:10:43.984546 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:43.979259 29439 raft_consensus.cc:504] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:43.988317 25981 leader_election.cc:302] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:43.952178 29330 raft_consensus.cc:3037] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:43.945533 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:43.970450 26046 rpcz_store.cc:276] 0807 11:10:42.442100 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:42.475382 (+ 33282us) service_pool.cc:227] Handling call
0807 11:10:42.475766 (+ 384us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:43.562755 (+1086989us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:43.936311 (+373556us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540022b354. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:43.936501 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2498461056}
I20260807 11:10:43.966732 29244 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:44.022740 25931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 222) took 5659 ms. Trace:
W20260807 11:10:44.022001 25801 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1486) took 3057 ms (client timeout 1603 ms). Trace:
W20260807 11:10:44.039702 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 51496.3us
I20260807 11:10:44.039301 25931 rpcz_store.cc:276] 0807 11:10:38.363378 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:38.363890 (+ 512us) service_pool.cc:227] Handling call
0807 11:10:38.412596 (+ 48706us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:42.886310 (+4473714us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.022430 (+1136120us) spinlock_profiling.cc:243] Waited 4.45 s on lock 0x7b54000361d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:44.022646 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10246031616}
I20260807 11:10:44.040980 25981 leader_election.cc:302] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:44.034844 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:44.036432 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:44.016242 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:44.011353 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:44.042488 25932 raft_consensus.cc:2379] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 7: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
W20260807 11:10:44.023312 26038 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43246 (request call id 375) took 1919 ms (client timeout 2130 ms). Trace:
W20260807 11:10:44.012243 29468 log.cc:924] Time spent T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.117s user 0.000s sys 0.001s
W20260807 11:10:44.043032 26038 rpcz_store.cc:269] 0807 11:10:42.103633 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:42.118241 (+ 14608us) service_pool.cc:227] Handling call
0807 11:10:44.023067 (+1904826us) spinlock_profiling.cc:243] Waited 498 ms on lock 0x7b540038de54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:44.023237 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1145095936}
W20260807 11:10:44.043476 25801 rpcz_store.cc:269] 0807 11:10:40.964526 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:40.964714 (+ 188us) service_pool.cc:227] Handling call
0807 11:10:44.021788 (+3057074us) spinlock_profiling.cc:243] Waited 2.06 s on lock 0x7b540014f5d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:44.021921 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4734217984}
I20260807 11:10:43.979521 29318 raft_consensus.cc:686] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:44.038735 29441 raft_consensus.cc:686] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 5 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:44.062232 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:44.065109 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:44.042760 26041 raft_consensus.cc:2379] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 7: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:44.063351 25981 leader_election.cc:302] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:10:44.030822 29303 consensus_meta.cc:287] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.532s user 0.009s sys 0.000s
I20260807 11:10:44.066658 25981 leader_election.cc:302] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:44.080188 29295 raft_consensus.cc:482] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:44.080705 29295 raft_consensus.cc:504] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:44.082818 29295 leader_election.cc:288] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:44.083760 29295 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:44.084448 29295 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:44.086974 29295 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:44.088064 29206 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:44.088697 29206 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:44.091063 29206 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:44.088682 29295 raft_consensus.cc:482] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:44.092900 29295 raft_consensus.cc:504] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:44.026201 29196 leader_election.cc:288] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:44.095443 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:44.021364 29417 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:44.013947 29179 leader_election.cc:288] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:44.068378 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:44.068750 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:44.030490 29410 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:44.015815 29250 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:44.019956 29307 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:44.039053 29001 consensus_queue.cc:260] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:44.094477 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:44.094480 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:44.067957 26037 raft_consensus.cc:2454] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:44.022634 29445 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:44.015017 29283 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:44.017627 29439 leader_election.cc:288] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:44.064741 28946 consensus_meta.cc:287] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.646s user 0.006s sys 0.006s
I20260807 11:10:44.061398 29272 consensus_queue.cc:1035] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.073s
I20260807 11:10:44.150521 25801 raft_consensus.cc:2379] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 4: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:44.114454 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:44.061546 29358 raft_consensus.cc:2726] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:44.101828 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:44.133337 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:44.151381 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:44.047415 29223 leader_election.cc:288] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:44.024866 29321 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:44.152384 26046 raft_consensus.cc:2454] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:44.153185 25872 leader_election.cc:302] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:44.120415 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:44.045580 26975 raft_consensus.cc:3037] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:44.194684 25752 leader_election.cc:302] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:44.210469 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:44.152052 26042 raft_consensus.cc:2454] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:44.205503 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:44.215057 25925 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:10:44.215553 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35276 (request call id 248) took 1318 ms. Trace:
I20260807 11:10:44.039776 29173 raft_consensus.cc:482] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:44.210708 25752 leader_election.cc:302] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:44.209123 25932 raft_consensus.cc:2454] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 5.
W20260807 11:10:44.211619 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 58555.7us
I20260807 11:10:44.198889 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:44.215950 25925 rpcz_store.cc:276] 0807 11:10:42.896882 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.205204 (+1308322us) service_pool.cc:227] Handling call
0807 11:10:44.215499 (+ 10295us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.215966 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:44.250743 26037 raft_consensus.cc:3037] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:44.251159 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:44.099538 29347 raft_consensus.cc:2781] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260807 11:10:44.117491 29472 log.cc:924] Time spent T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.132s user 0.001s sys 0.000s
I20260807 11:10:44.217703 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:44.074011 29266 raft_consensus.cc:2781] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:44.047938 29479 consensus_queue.cc:1035] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.151s
I20260807 11:10:44.218714 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:44.098860 29029 raft_consensus.cc:2781] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:44.223178 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:44.152050 28946 raft_consensus.cc:2726] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:44.229266 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:10:44.262807 25926 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35276 (request call id 237) took 2550 ms (client timeout 3139 ms). Trace:
I20260807 11:10:44.263116 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:44.263317 25926 rpcz_store.cc:269] 0807 11:10:41.712031 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:42.530773 (+818742us) service_pool.cc:227] Handling call
0807 11:10:44.262620 (+1731847us) spinlock_profiling.cc:243] Waited 989 ms on lock 0x7b54003b1fd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:44.262751 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2275687168}
I20260807 11:10:44.263815 25932 raft_consensus.cc:2454] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:44.264387 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1659) took 1395 ms. Trace:
I20260807 11:10:44.265954 29266 raft_consensus.cc:686] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:44.134003 29441 consensus_queue.cc:237] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:44.267202 29266 consensus_queue.cc:237] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:44.126148 29295 leader_election.cc:288] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:44.122498 29352 leader_election.cc:288] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:44.138355 29254 consensus_queue.cc:1046] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.084s
I20260807 11:10:44.216557 29223 raft_consensus.cc:2726] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 11:10:44.218928 29001 raft_consensus.cc:3037] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
W20260807 11:10:44.236064 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 195181us
I20260807 11:10:44.221277 29283 raft_consensus.cc:2781] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:44.198973 29410 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:44.241233 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:44.242954 25800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 303) took 1487 ms. Trace:
I20260807 11:10:44.104347 29303 raft_consensus.cc:504] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:44.108711 29440 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:44.110994 29397 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:44.251569 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
W20260807 11:10:44.254880 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 44299us
I20260807 11:10:44.363384 25752 leader_election.cc:302] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:44.254942 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:44.365039 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:44.366369 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:44.199124 29445 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:44.262849 26046 raft_consensus.cc:3037] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:44.264753 25932 rpcz_store.cc:276] 0807 11:10:42.869382 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.257874 (+1388492us) service_pool.cc:227] Handling call
0807 11:10:44.264329 (+ 6455us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.110392 29318 consensus_queue.cc:237] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:44.264559 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:44.260751 29315 raft_consensus.cc:2781] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:44.127592 29278 consensus_queue.cc:260] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:44.323865 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:44.309893 26041 raft_consensus.cc:2379] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 4: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:44.313843 26037 raft_consensus.cc:2454] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:44.395881 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:44.255055 29347 raft_consensus.cc:482] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:44.401700 26986 raft_consensus.cc:2379] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 5: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:44.309617 25800 rpcz_store.cc:276] 0807 11:10:42.756008 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:43.899237 (+1143229us) service_pool.cc:227] Handling call
0807 11:10:43.902774 (+ 3537us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:43.902984 (+ 210us) raft_consensus.cc:1520] Early marking committed up to index 6
0807 11:10:44.048185 (+145201us) log.cc:842] Serialized 14 byte log entry
0807 11:10:44.242367 (+194182us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:44.242465 (+ 98us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:10:44.242539 (+ 74us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.242672 (+ 133us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:44.242864 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":315810,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":3596,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":5676,"thread_start_us":335446,"threads_started":2}
I20260807 11:10:44.235576 29334 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:44.270395 29029 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:44.270958 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:44.349560 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:44.399436 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:44.501978 26046 raft_consensus.cc:2454] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:10:44.350598 29254 consensus_queue.cc:1225] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer b22d69db45194908ad3d345eba9badd9 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
W20260807 11:10:44.499397 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 128398us
W20260807 11:10:44.498520 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 202800us
I20260807 11:10:44.499408 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:44.509876 26041 raft_consensus.cc:2454] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:44.510089 25752 leader_election.cc:302] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:44.506420 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:44.515415 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:44.516021 25801 raft_consensus.cc:2396] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260807 11:10:44.412088 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:44.502463 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:44.431392 29244 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:44.473356 29440 raft_consensus.cc:482] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:44.510160 25869 leader_election.cc:302] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:44.506119 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:44.511322 26041 raft_consensus.cc:1226] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260807 11:10:44.505141 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:44.360090 29283 raft_consensus.cc:686] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:44.389405 29315 raft_consensus.cc:482] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:44.517776 25926 raft_consensus.cc:2379] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 5: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:44.522995 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1635) took 2028 ms. Trace:
I20260807 11:10:44.442317 29278 raft_consensus.cc:3037] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:44.517040 25932 raft_consensus.cc:3037] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:44.401369 29223 raft_consensus.cc:2781] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:44.525604 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:44.404227 29347 raft_consensus.cc:3037] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:44.527933 26041 rpcz_store.cc:276] 0807 11:10:42.494803 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.510623 (+2015820us) service_pool.cc:227] Handling call
0807 11:10:44.511043 (+ 420us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:44.522609 (+ 11566us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.522938 (+ 329us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.523725 25869 leader_election.cc:302] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:44.529726 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1648) took 2903 ms. Trace:
I20260807 11:10:44.530308 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:44.532615 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:44.538713 25925 rpcz_store.cc:276] 0807 11:10:41.626775 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.390682 (+2763907us) service_pool.cc:227] Handling call
0807 11:10:44.529667 (+138985us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.514547 29029 raft_consensus.cc:3037] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:44.532888 29315 raft_consensus.cc:3037] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:44.537533 25799 raft_consensus.cc:1226] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 6. Current term is 7. Ops: []
I20260807 11:10:44.552284 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:44.261541 29173 raft_consensus.cc:504] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:44.534636 26038 raft_consensus.cc:2454] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:44.539700 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59312 (request call id 1641) took 1269 ms. Trace:
I20260807 11:10:44.526194 29254 raft_consensus.cc:2726] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:44.539186 25926 raft_consensus.cc:2454] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:44.558351 25926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1665) took 1346 ms. Trace:
I20260807 11:10:44.558743 25926 rpcz_store.cc:276] 0807 11:10:43.211659 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.532305 (+1320646us) service_pool.cc:227] Handling call
0807 11:10:44.558254 (+ 25949us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.559768 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:44.549340 26046 raft_consensus.cc:1226] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:44.560534 25926 raft_consensus.cc:2454] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 5.
I20260807 11:10:44.553525 25925 raft_consensus.cc:2396] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260807 11:10:44.534085 29223 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:44.537982 29191 raft_consensus.cc:2781] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:10:44.557174 26042 rpcz_store.cc:276] 0807 11:10:43.270062 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.516842 (+1246780us) service_pool.cc:227] Handling call
0807 11:10:44.539638 (+ 22796us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.557650 25872 leader_election.cc:302] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:44.526993 29302 raft_consensus.cc:3037] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:44.561321 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1638) took 1917 ms. Trace:
I20260807 11:10:44.561399 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:44.561731 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1559) took 1661 ms. Trace:
I20260807 11:10:44.565380 25926 raft_consensus.cc:3037] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:44.535367 25752 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:44.565656 25925 rpcz_store.cc:276] 0807 11:10:42.900408 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.552020 (+1651612us) service_pool.cc:227] Handling call
0807 11:10:44.561660 (+ 9640us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.553117 29179 raft_consensus.cc:2726] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:44.569236 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:44.529345 29440 raft_consensus.cc:504] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:44.570104 25752 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:44.568025 26046 rpcz_store.cc:276] 0807 11:10:42.643927 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.527115 (+1883188us) service_pool.cc:227] Handling call
0807 11:10:44.540642 (+ 13527us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:44.561105 (+ 20463us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.561273 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.572700 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1749) took 1792 ms. Trace:
I20260807 11:10:44.575366 26038 rpcz_store.cc:276] 0807 11:10:42.780590 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.559590 (+1779000us) service_pool.cc:227] Handling call
0807 11:10:44.572051 (+ 12461us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:44.572296 (+ 245us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:44.572332 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:44.572388 (+ 56us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:44.572429 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.572496 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:44.572617 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.578127 26042 raft_consensus.cc:1226] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:44.578135 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:44.578936 25925 raft_consensus.cc:2454] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:44.547367 29479 consensus_queue.cc:1046] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260807 11:10:44.562667 29223 raft_consensus.cc:3037] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:44.579762 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:44.583784 29397 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:44.584924 25932 raft_consensus.cc:2454] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:44.585834 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:44.586423 25932 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:44.592351 29253 raft_consensus.cc:3037] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:44.594493 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1750) took 1813 ms. Trace:
I20260807 11:10:44.594888 26042 rpcz_store.cc:276] 0807 11:10:42.780873 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.565347 (+1784474us) service_pool.cc:227] Handling call
0807 11:10:44.577964 (+ 12617us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:44.594215 (+ 16251us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.594434 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.583256 25925 raft_consensus.cc:2454] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:44.596443 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:44.599789 29445 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:44.600154 28797 raft_consensus.cc:3037] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:44.601384 29173 leader_election.cc:288] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:44.568962 29244 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:44.605659 26974 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 417) took 2855 ms (client timeout 3033 ms). Trace:
W20260807 11:10:44.606107 26974 rpcz_store.cc:269] 0807 11:10:41.750623 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:41.753308 (+ 2685us) service_pool.cc:227] Handling call
0807 11:10:44.605433 (+2852125us) spinlock_profiling.cc:243] Waited 338 ms on lock 0x7b5400470754. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:44.605576 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":778254080}
I20260807 11:10:44.607358 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:44.607632 25750 leader_election.cc:398] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 5. Current term is 6.
I20260807 11:10:44.608106 25750 leader_election.cc:401] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
W20260807 11:10:44.609117 29486 log.cc:924] Time spent T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.207s user 0.001s sys 0.000s
W20260807 11:10:44.610445 25918 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1533) took 2972 ms (client timeout 2008 ms). Trace:
W20260807 11:10:44.610817 25918 rpcz_store.cc:269] 0807 11:10:41.638045 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:42.047001 (+408956us) service_pool.cc:227] Handling call
0807 11:10:44.610236 (+2563235us) spinlock_profiling.cc:243] Waited 341 ms on lock 0x7b54001df954. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:44.610352 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":783551360}
I20260807 11:10:44.611375 25786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1633) took 2142 ms. Trace:
I20260807 11:10:44.611807 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:44.611702 25786 rpcz_store.cc:276] 0807 11:10:42.468723 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:42.472885 (+ 4162us) service_pool.cc:227] Handling call
0807 11:10:42.473322 (+ 437us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:43.969544 (+1496222us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.611153 (+641609us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b54000e4cd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:44.611322 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3395669376}
I20260807 11:10:44.612674 25918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1560) took 1002 ms. Trace:
I20260807 11:10:44.612915 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:44.613039 25918 rpcz_store.cc:276] 0807 11:10:43.609825 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.611537 (+1001712us) service_pool.cc:227] Handling call
0807 11:10:44.612625 (+ 1088us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.613649 25786 raft_consensus.cc:2379] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 5: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:44.593814 29191 raft_consensus.cc:686] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:44.613912 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:44.614509 25918 raft_consensus.cc:3037] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:44.614998 25786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 310) took 1306 ms. Trace:
I20260807 11:10:44.614847 29191 consensus_queue.cc:237] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:44.615355 25786 rpcz_store.cc:276] 0807 11:10:43.308325 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.614253 (+1305928us) service_pool.cc:227] Handling call
0807 11:10:44.614559 (+ 306us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:44.614711 (+ 152us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:44.614737 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:44.614774 (+ 37us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:44.614814 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.614860 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:44.614955 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.568416 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:44.572299 29321 raft_consensus.cc:2781] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:10:44.595590 25926 raft_consensus.cc:2454] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 5.
I20260807 11:10:44.620389 25981 leader_election.cc:302] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
W20260807 11:10:44.593336 29484 log.cc:924] Time spent T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.333s user 0.000s sys 0.001s
I20260807 11:10:44.572913 29490 raft_consensus.cc:2781] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:44.620412 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:44.622284 25981 leader_election.cc:302] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:44.618558 26038 raft_consensus.cc:1226] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:44.583562 29334 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:44.625038 25932 raft_consensus.cc:2454] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 7.
I20260807 11:10:44.626211 25800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 307) took 1736 ms. Trace:
I20260807 11:10:44.629199 25786 raft_consensus.cc:1226] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 6. Current term is 7. Ops: []
I20260807 11:10:44.626768 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:44.631223 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 384) took 1814 ms. Trace:
I20260807 11:10:44.582317 29303 leader_election.cc:288] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:44.608621 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:44.624891 25925 raft_consensus.cc:1226] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: [3.2-4.3]
I20260807 11:10:44.632315 25926 raft_consensus.cc:2379] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 5: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:44.635387 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1637) took 3017 ms. Trace:
I20260807 11:10:44.632335 25981 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:44.636125 25750 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:44.633163 25800 rpcz_store.cc:276] 0807 11:10:42.889448 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.526575 (+1637127us) service_pool.cc:227] Handling call
0807 11:10:44.527341 (+ 766us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:44.527679 (+ 338us) raft_consensus.cc:1520] Early marking committed up to index 7
0807 11:10:44.527939 (+ 260us) log.cc:842] Serialized 14 byte log entry
0807 11:10:44.625888 (+ 97949us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:44.625976 (+ 88us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:10:44.626023 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.626082 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:44.626167 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":3164,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":4135,"thread_start_us":97572,"threads_started":1}
I20260807 11:10:44.635763 25925 rpcz_store.cc:276] 0807 11:10:41.617882 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.624042 (+3006160us) service_pool.cc:227] Handling call
0807 11:10:44.624364 (+ 322us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:10:44.635035 (+ 10671us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.635323 (+ 288us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:10:44.634644 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 64592.2us
I20260807 11:10:44.633700 25786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53324 (request call id 1513) took 1024 ms. Trace:
I20260807 11:10:44.641288 25786 rpcz_store.cc:276] 0807 11:10:43.609183 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.616455 (+1007272us) service_pool.cc:227] Handling call
0807 11:10:44.628960 (+ 12505us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:44.633374 (+ 4414us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.633631 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.634295 26042 raft_consensus.cc:3037] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 5
I20260807 11:10:44.619369 29397 raft_consensus.cc:482] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:44.643329 25750 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:44.644629 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:44.634306 26038 rpcz_store.cc:276] 0807 11:10:42.817266 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.576646 (+1759380us) service_pool.cc:227] Handling call
0807 11:10:44.618401 (+ 41755us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:44.630970 (+ 12569us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.631176 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.645614 25921 raft_consensus.cc:2379] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 7: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:44.642140 25981 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:44.646205 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35276 (request call id 247) took 1749 ms. Trace:
I20260807 11:10:44.641633 25925 raft_consensus.cc:1226] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:44.646701 25921 rpcz_store.cc:276] 0807 11:10:42.896556 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.623295 (+1726739us) service_pool.cc:227] Handling call
0807 11:10:44.646149 (+ 22854us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:10:44.648262 25806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1649) took 1979 ms (client timeout 2131 ms). Trace:
I20260807 11:10:44.649087 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1643) took 3023 ms. Trace:
I20260807 11:10:44.649488 25925 rpcz_store.cc:276] 0807 11:10:41.625268 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.638578 (+3013310us) service_pool.cc:227] Handling call
0807 11:10:44.641454 (+ 2876us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:44.646950 (+ 5496us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.647147 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.648147 25981 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:44.633922 25932 raft_consensus.cc:1226] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260807 11:10:44.648782 25806 rpcz_store.cc:269] 0807 11:10:42.668472 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:42.833029 (+164557us) service_pool.cc:227] Handling call
0807 11:10:44.648078 (+1815049us) spinlock_profiling.cc:243] Waited 714 ms on lock 0x7b54002f6f54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:44.648198 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1642059520}
I20260807 11:10:44.651618 28946 raft_consensus.cc:2781] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:44.652361 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1638) took 3032 ms. Trace:
I20260807 11:10:44.630158 29490 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:44.653491 29490 raft_consensus.cc:3037] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:44.631477 29441 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:44.654685 29006 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:44.653201 29440 leader_election.cc:288] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:44.655827 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1628) took 2715 ms. Trace:
I20260807 11:10:44.656287 26039 rpcz_store.cc:276] 0807 11:10:41.940396 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:42.413251 (+472855us) service_pool.cc:227] Handling call
0807 11:10:42.421579 (+ 8328us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:43.030245 (+608666us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.655570 (+1625325us) spinlock_profiling.cc:243] Waited 608 ms on lock 0x7b5400313654. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:44.655752 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1397959936}
I20260807 11:10:44.657663 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:44.658295 26039 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:44.644105 29129 raft_consensus.cc:2781] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:44.663014 29129 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:44.663501 29129 raft_consensus.cc:3037] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:44.652999 25932 rpcz_store.cc:276] 0807 11:10:41.620247 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.633390 (+3013143us) service_pool.cc:227] Handling call
0807 11:10:44.633712 (+ 322us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:44.651901 (+ 18189us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.652306 (+ 405us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.654528 29441 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:44.681856 25800 raft_consensus.cc:1226] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:44.665378 29397 raft_consensus.cc:504] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:44.690390 25981 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:44.632606 29321 raft_consensus.cc:686] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:44.639281 29307 raft_consensus.cc:2781] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:44.594072 29283 consensus_queue.cc:237] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:44.647921 29352 raft_consensus.cc:2781] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:44.698583 25869 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:44.731595 25869 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:44.630797 26975 raft_consensus.cc:2454] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:44.733472 25869 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:44.699644 25800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53324 (request call id 1515) took 1085 ms. Trace:
I20260807 11:10:44.646255 29246 raft_consensus.cc:2726] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 5. Current term is 6.
I20260807 11:10:44.652283 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:44.702471 25806 raft_consensus.cc:1226] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:44.624348 29169 consensus_queue.cc:1046] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.067s
I20260807 11:10:44.690894 29440 raft_consensus.cc:2726] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:44.718587 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:44.696522 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 53262.7us
I20260807 11:10:44.747201 25750 leader_election.cc:302] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:44.721423 29006 raft_consensus.cc:2781] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:10:44.700165 25917 raft_consensus.cc:2454] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:44.720875 26033 raft_consensus.cc:2454] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:44.721794 29307 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:44.708323 25981 leader_election.cc:302] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:44.734547 25800 rpcz_store.cc:276] 0807 11:10:43.614122 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.638115 (+1023993us) service_pool.cc:227] Handling call
0807 11:10:44.649225 (+ 11110us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:44.699400 (+ 50175us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.699589 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.736383 26042 raft_consensus.cc:2454] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:10:44.737421 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:44.755509 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:44.755851 26938 leader_election.cc:302] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:10:44.743872 29454 consensus_meta.cc:287] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.805s user 0.013s sys 0.000s
I20260807 11:10:44.739398 29198 raft_consensus.cc:3032] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:44.691406 29373 raft_consensus.cc:2781] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:44.743342 25806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53324 (request call id 1521) took 1120 ms. Trace:
I20260807 11:10:44.729065 29410 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:44.703418 29277 consensus_meta.cc:287] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.664s user 0.010s sys 0.000s
I20260807 11:10:44.709105 25918 raft_consensus.cc:2454] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:10:44.750705 25750 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:44.702471 25921 raft_consensus.cc:1226] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:44.728281 29295 raft_consensus.cc:2781] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260807 11:10:44.739744 28449 consensus_meta.cc:287] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.786s user 0.009s sys 0.002s
I20260807 11:10:44.725103 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1646) took 3099 ms. Trace:
I20260807 11:10:44.729853 29352 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:44.757053 29307 raft_consensus.cc:3037] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
W20260807 11:10:44.744884 29330 consensus_meta.cc:287] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.700s user 0.010s sys 0.000s
I20260807 11:10:44.738677 25869 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:44.758097 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:10:44.758432 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 50191.7us
I20260807 11:10:44.757738 26042 raft_consensus.cc:2379] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 5: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:44.759963 26038 raft_consensus.cc:2379] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 4: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:44.701761 28761 consensus_queue.cc:1046] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20260807 11:10:44.762024 25800 raft_consensus.cc:1226] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:44.761862 25806 rpcz_store.cc:276] 0807 11:10:43.622615 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.651234 (+1028619us) service_pool.cc:227] Handling call
0807 11:10:44.702303 (+ 51069us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:44.743148 (+ 40845us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.743299 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.758509 29006 raft_consensus.cc:686] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:44.759835 29198 raft_consensus.cc:729] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:44.752879 29246 raft_consensus.cc:2726] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 11:10:44.753615 29505 raft_consensus.cc:2726] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:44.745833 29254 raft_consensus.cc:2726] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:44.752375 29123 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:44.752377 28438 raft_consensus.cc:482] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:44.811823 25800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53324 (request call id 1522) took 1187 ms. Trace:
I20260807 11:10:44.812300 25800 rpcz_store.cc:276] 0807 11:10:43.624909 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.755002 (+1130093us) service_pool.cc:227] Handling call
0807 11:10:44.760712 (+ 5710us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:44.809114 (+ 48402us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.811751 (+ 2637us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.770699 25932 rpcz_store.cc:276] 0807 11:10:41.626159 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.682503 (+3056344us) service_pool.cc:227] Handling call
0807 11:10:44.718247 (+ 35744us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:44.724727 (+ 6480us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:44.724797 (+ 70us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:44.724854 (+ 57us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:44.724897 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.724947 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:44.725051 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.814411 25800 raft_consensus.cc:1226] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:44.772964 25750 leader_election.cc:302] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:44.763636 29194 raft_consensus.cc:2781] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:10:44.740281 28797 raft_consensus.cc:2726] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:44.776283 26985 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 7.
I20260807 11:10:44.829119 25806 raft_consensus.cc:1226] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:44.782548 29205 raft_consensus.cc:3037] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:44.764240 29440 raft_consensus.cc:2781] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:44.805297 25981 leader_election.cc:302] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:10:44.805749 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 67137.9us
I20260807 11:10:44.837001 25869 leader_election.cc:302] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:44.786952 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:44.764120 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1644) took 3138 ms. Trace:
I20260807 11:10:44.763079 29321 consensus_queue.cc:237] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:44.761821 29373 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:44.810469 26039 raft_consensus.cc:3037] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:44.769913 29295 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:44.816241 25750 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
W20260807 11:10:44.868201 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 52060.7us
I20260807 11:10:44.777601 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:44.754343 28946 raft_consensus.cc:686] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:44.768334 29277 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:44.777508 29418 raft_consensus.cc:2726] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:44.839556 29440 raft_consensus.cc:482] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:44.778316 29352 raft_consensus.cc:3037] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
W20260807 11:10:44.779942 29496 log.cc:924] Time spent T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.186s user 0.000s sys 0.001s
I20260807 11:10:44.761441 29454 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:44.831187 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 386) took 1738 ms. Trace:
I20260807 11:10:44.899611 26038 rpcz_store.cc:276] 0807 11:10:43.092465 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.806479 (+1714014us) service_pool.cc:227] Handling call
0807 11:10:44.830569 (+ 24090us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:44.830789 (+ 220us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:44.830825 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:44.830877 (+ 52us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:10:44.830918 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.830984 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:44.831118 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.900501 25932 raft_consensus.cc:1226] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:44.834821 25800 raft_consensus.cc:1261] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260807 11:10:44.825928 28438 raft_consensus.cc:504] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:44.806653 29330 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:44.826719 29001 consensus_meta.cc:287] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.505s user 0.009s sys 0.000s
I20260807 11:10:44.813655 29441 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:10:44.907051 25916 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35276 (request call id 245) took 2292 ms (client timeout 1619 ms). Trace:
I20260807 11:10:44.907441 26987 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 425) took 1914 ms. Trace:
I20260807 11:10:44.814438 29123 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:44.852859 28797 raft_consensus.cc:3037] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:44.907555 25925 raft_consensus.cc:1226] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 5. Current term is 6. Ops: []
I20260807 11:10:44.907959 26987 rpcz_store.cc:276] 0807 11:10:42.993009 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:42.993197 (+ 188us) service_pool.cc:227] Handling call
0807 11:10:43.056941 (+ 63744us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:43.571924 (+514983us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.907208 (+1335284us) spinlock_profiling.cc:243] Waited 514 ms on lock 0x7b54005fe954. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:44.907386 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1182758272}
I20260807 11:10:44.897360 26975 raft_consensus.cc:2379] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 6: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:44.845188 29302 raft_consensus.cc:2781] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:10:44.919631 25920 raft_consensus.cc:2396] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 11:10:44.873010 25750 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:44.920280 25918 raft_consensus.cc:1226] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:44.920186 25797 raft_consensus.cc:1226] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 4. Current term is 5. Ops: []
W20260807 11:10:44.920377 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 47466.5us
I20260807 11:10:44.921700 29321 raft_consensus.cc:2781] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:10:44.922292 29321 raft_consensus.cc:686] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:44.923161 29321 consensus_queue.cc:237] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:44.828503 29198 consensus_queue.cc:260] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:44.843726 25921 rpcz_store.cc:276] 0807 11:10:41.625554 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.647432 (+3021878us) service_pool.cc:227] Handling call
0807 11:10:44.702303 (+ 54871us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:44.763872 (+ 61569us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.764062 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.842670 29194 raft_consensus.cc:686] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:44.870430 29373 raft_consensus.cc:3037] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:44.931253 26039 raft_consensus.cc:2454] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:10:44.872252 29295 raft_consensus.cc:3037] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:44.843667 25869 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:44.935426 26987 raft_consensus.cc:1226] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20260807 11:10:44.935909 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 92448.6us
I20260807 11:10:44.936210 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:44.903190 25931 raft_consensus.cc:2379] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 5: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:44.874246 29316 raft_consensus.cc:2781] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 11:10:44.842077 29254 consensus_queue.cc:1046] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20260807 11:10:44.904011 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1527) took 3276 ms. Trace:
I20260807 11:10:44.937348 25796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1514) took 1327 ms. Trace:
W20260807 11:10:44.907613 25916 rpcz_store.cc:269] 0807 11:10:42.614792 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:42.641970 (+ 27178us) service_pool.cc:227] Handling call
0807 11:10:44.906778 (+2264808us) spinlock_profiling.cc:243] Waited 787 ms on lock 0x7b54001df454. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:44.906960 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1810283904}
I20260807 11:10:44.809504 29179 raft_consensus.cc:2726] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:44.895303 29440 raft_consensus.cc:3037] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:44.918901 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 387) took 1826 ms. Trace:
I20260807 11:10:44.942270 26038 rpcz_store.cc:276] 0807 11:10:43.092826 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.901052 (+1808226us) service_pool.cc:227] Handling call
0807 11:10:44.918411 (+ 17359us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:44.918589 (+ 178us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:44.918612 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:44.918657 (+ 45us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:10:44.918683 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.918733 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:44.918842 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.826186 29246 raft_consensus.cc:2781] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:44.926663 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:44.924741 25750 leader_election.cc:302] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:44.876421 29510 consensus_queue.cc:1046] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20260807 11:10:44.931082 25919 raft_consensus.cc:2379] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 5: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:44.880394 29334 consensus_queue.cc:1046] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260807 11:10:44.902599 29397 leader_election.cc:288] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:44.933995 26039 raft_consensus.cc:1226] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 5. Current term is 6. Ops: []
I20260807 11:10:44.955595 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 389) took 1862 ms. Trace:
I20260807 11:10:44.955911 26039 rpcz_store.cc:276] 0807 11:10:43.095532 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.933416 (+1837884us) service_pool.cc:227] Handling call
0807 11:10:44.933780 (+ 364us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:44.955351 (+ 21571us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.955541 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.865687 29006 consensus_queue.cc:237] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:44.940330 25932 rpcz_store.cc:276] 0807 11:10:41.627722 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.844628 (+3216906us) service_pool.cc:227] Handling call
0807 11:10:44.896318 (+ 51690us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:44.901029 (+ 4711us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.903932 (+ 2903us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.941308 25796 rpcz_store.cc:276] 0807 11:10:43.609494 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.906347 (+1296853us) service_pool.cc:227] Handling call
0807 11:10:44.937276 (+ 30929us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:10:44.903079 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 230574us
I20260807 11:10:44.919126 25803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53324 (request call id 1503) took 2495 ms. Trace:
I20260807 11:10:44.946671 25869 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:44.946053 25921 raft_consensus.cc:2454] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:44.953917 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1751) took 2172 ms. Trace:
I20260807 11:10:44.962080 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:44.962244 25869 leader_election.cc:302] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:44.963732 25869 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:44.965587 25869 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:44.958966 26046 rpcz_store.cc:276] 0807 11:10:42.781107 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.572356 (+1791249us) service_pool.cc:227] Handling call
0807 11:10:44.598159 (+ 25803us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:44.598674 (+ 515us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 11:10:44.759420 (+160746us) log.cc:842] Serialized 14 byte log entry
0807 11:10:44.953586 (+194166us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:44.953632 (+ 46us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:44.953678 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.953757 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:44.953865 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":263474,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":1038,"thread_start_us":323717,"threads_started":3}
I20260807 11:10:44.934273 29302 raft_consensus.cc:686] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 5 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:44.966823 29493 raft_consensus.cc:2781] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:44.943703 29169 consensus_queue.cc:1046] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.030s
I20260807 11:10:44.957885 25803 rpcz_store.cc:276] 0807 11:10:42.424197 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:42.425020 (+ 823us) service_pool.cc:227] Handling call
0807 11:10:42.425382 (+ 362us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:10:42.491713 (+ 66331us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:42.491742 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 11:10:42.503315 (+ 11573us) log.cc:842] Serialized 52 byte log entry
0807 11:10:42.753607 (+250292us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:42.772874 (+ 19267us) log.cc:842] Serialized 14 byte log entry
0807 11:10:42.788061 (+ 15187us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.906836 (+2118775us) spinlock_profiling.cc:243] Waited 52.3 ms on lock 0x7b54002f5154. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:44.906884 (+ 48us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:44.906934 (+ 50us) raft_consensus.cc:1671] finished
0807 11:10:44.918932 (+ 11998us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:44.919073 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28559,"prepare.queue_time_us":22075,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":34205,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":25552,"spinlock_wait_cycles":139754624,"thread_start_us":174804,"threads_started":1,"wal-append.queue_time_us":319470,"wal-append.run_cpu_time_us":4313,"wal-append.run_wall_time_us":1537766}
I20260807 11:10:44.983235 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:44.945708 29318 raft_consensus.cc:482] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
W20260807 11:10:44.954905 29501 log.cc:924] Time spent T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.112s user 0.000s sys 0.001s
W20260807 11:10:44.973867 25808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1646) took 2334 ms (client timeout 1851 ms). Trace:
W20260807 11:10:44.989148 25808 rpcz_store.cc:269] 0807 11:10:42.639991 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:42.658012 (+ 18021us) service_pool.cc:227] Handling call
0807 11:10:44.973686 (+2315674us) spinlock_profiling.cc:243] Waited 48.2 ms on lock 0x7b54002d4bd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:44.973802 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":110940928}
I20260807 11:10:44.974970 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 393) took 1097 ms. Trace:
I20260807 11:10:44.990339 26039 rpcz_store.cc:276] 0807 11:10:43.877561 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.969162 (+1091601us) service_pool.cc:227] Handling call
0807 11:10:44.974508 (+ 5346us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:44.974676 (+ 168us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:44.974698 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:44.974747 (+ 49us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:10:44.974775 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.974825 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:44.974919 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.983361 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:44.942890 29198 raft_consensus.cc:3037] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:44.983896 25932 raft_consensus.cc:2454] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:44.991706 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:44.984606 25750 leader_election.cc:302] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:44.986634 25916 raft_consensus.cc:1226] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:44.961462 29316 raft_consensus.cc:686] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 7 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:44.939671 29001 raft_consensus.cc:2741] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260807 11:10:44.992120 25921 raft_consensus.cc:2454] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 5.
I20260807 11:10:44.978348 29179 raft_consensus.cc:2726] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 11:10:44.977871 29505 raft_consensus.cc:3037] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:44.954615 29410 consensus_queue.cc:1035] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20260807 11:10:44.968868 29246 raft_consensus.cc:686] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:44.995097 25869 leader_election.cc:302] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:44.972224 29196 raft_consensus.cc:2781] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:44.973230 29283 raft_consensus.cc:2726] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 11:10:45.007848 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:45.009100 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:45.010020 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:45.011695 29490 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:45.014748 29315 raft_consensus.cc:504] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:45.017509 29515 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:45.018085 29515 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:45.020597 29273 raft_consensus.cc:482] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:44.944265 28946 consensus_queue.cc:237] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:45.021162 29273 raft_consensus.cc:504] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:45.024282 29273 leader_election.cc:288] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:45.024782 29244 raft_consensus.cc:3032] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:45.024948 29454 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:45.025198 29244 raft_consensus.cc:729] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:44.969305 29500 consensus_queue.cc:1046] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.053s
I20260807 11:10:45.026144 29244 consensus_queue.cc:260] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:45.027161 29244 raft_consensus.cc:3037] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:45.027068 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:45.027753 25796 raft_consensus.cc:3037] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:45.029152 25918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1647) took 3402 ms. Trace:
I20260807 11:10:45.029506 25918 rpcz_store.cc:276] 0807 11:10:41.626438 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.765794 (+3139356us) service_pool.cc:227] Handling call
0807 11:10:44.766132 (+ 338us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:44.920881 (+154749us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:45.028937 (+108056us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b5400036bd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:45.029097 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":354106752}
I20260807 11:10:45.030818 25918 raft_consensus.cc:1226] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 5 LEADER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:45.031345 25918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1545) took 3389 ms. Trace:
I20260807 11:10:45.031618 25918 rpcz_store.cc:276] 0807 11:10:41.642419 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:45.030347 (+3387928us) service_pool.cc:227] Handling call
0807 11:10:45.030664 (+ 317us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:45.031139 (+ 475us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:45.031291 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:45.033227 29302 consensus_queue.cc:237] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:44.994129 25916 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1530) took 3365 ms. Trace:
I20260807 11:10:44.979115 29191 consensus_queue.cc:1046] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20260807 11:10:45.003966 25932 raft_consensus.cc:1226] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260807 11:10:44.970418 29098 consensus_queue.cc:1046] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20260807 11:10:44.985271 29254 raft_consensus.cc:2726] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:10:44.995298 29169 raft_consensus.cc:3032] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:45.037792 25918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 238) took 3318 ms. Trace:
I20260807 11:10:45.045197 25916 rpcz_store.cc:276] 0807 11:10:41.628634 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.963004 (+3334370us) service_pool.cc:227] Handling call
0807 11:10:44.986377 (+ 23373us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:44.993888 (+ 7511us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:44.994075 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:45.000923 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:45.136694 26039 raft_consensus.cc:2454] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:45.047605 25921 raft_consensus.cc:1226] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:45.138000 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1544) took 3495 ms. Trace:
I20260807 11:10:45.138352 25921 rpcz_store.cc:276] 0807 11:10:41.642089 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.996542 (+3354453us) service_pool.cc:227] Handling call
0807 11:10:45.035355 (+ 38813us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:45.137750 (+102395us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:45.137940 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:45.085649 26046 raft_consensus.cc:1226] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:45.105428 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1543) took 3463 ms. Trace:
I20260807 11:10:45.027089 29500 raft_consensus.cc:3032] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:45.020686 29515 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:45.051594 26974 raft_consensus.cc:1261] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260807 11:10:45.006698 29179 raft_consensus.cc:3037] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:45.139873 29415 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:45.172549 29415 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:45.173779 29315 leader_election.cc:288] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:45.133162 29440 raft_consensus.cc:504] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:45.174960 29415 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:45.176271 29440 leader_election.cc:288] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:45.176975 26042 raft_consensus.cc:1226] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
W20260807 11:10:45.178158 29347 consensus_meta.cc:287] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.635s user 0.010s sys 0.000s
I20260807 11:10:45.178606 29347 raft_consensus.cc:504] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:45.180888 26037 raft_consensus.cc:3037] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:45.058782 29410 consensus_queue.cc:1046] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.134s
I20260807 11:10:45.181983 29246 consensus_queue.cc:237] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:44.994979 29194 consensus_queue.cc:237] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:45.185336 26975 raft_consensus.cc:1226] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 6. Current term is 7. Ops: []
W20260807 11:10:45.189821 29223 consensus_meta.cc:287] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.573s user 0.009s sys 0.000s
I20260807 11:10:45.190248 29223 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:45.191275 25801 raft_consensus.cc:1226] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:45.192057 26039 raft_consensus.cc:1226] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:45.194303 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:45.194902 25916 raft_consensus.cc:3037] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:45.135578 29169 raft_consensus.cc:729] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:44.993654 29318 raft_consensus.cc:504] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:45.139411 25932 rpcz_store.cc:276] 0807 11:10:41.641773 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.992526 (+3350753us) service_pool.cc:227] Handling call
0807 11:10:45.003743 (+ 11217us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:45.105086 (+101343us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:45.105333 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:44.999078 29397 raft_consensus.cc:2781] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:45.051488 29253 raft_consensus.cc:482] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:45.223043 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1753) took 1338 ms. Trace:
I20260807 11:10:45.262976 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1755) took 1367 ms. Trace:
I20260807 11:10:45.263242 26046 rpcz_store.cc:276] 0807 11:10:43.884483 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:45.000441 (+1115958us) service_pool.cc:227] Handling call
0807 11:10:45.047898 (+ 47457us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:45.139318 (+ 91420us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:45.222969 (+ 83651us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:45.036696 29129 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:45.263396 26039 rpcz_store.cc:276] 0807 11:10:43.895827 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:45.137497 (+1241670us) service_pool.cc:227] Handling call
0807 11:10:45.191833 (+ 54336us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:45.262619 (+ 70786us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:45.262899 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:45.143932 29454 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:45.389151 28946 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:45.268513 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:45.264545 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:45.267724 29254 raft_consensus.cc:482] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:45.135950 25918 rpcz_store.cc:276] 0807 11:10:41.719109 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:45.032274 (+3313165us) service_pool.cc:227] Handling call
0807 11:10:45.032562 (+ 288us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:45.032738 (+ 176us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:45.037542 (+ 4804us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:45.037590 (+ 48us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:45.037616 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:45.037658 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:45.037751 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:45.430205 25752 leader_election.cc:302] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:45.267570 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 245932us
I20260807 11:10:45.141458 29277 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:45.143042 29500 raft_consensus.cc:729] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:45.156153 29196 raft_consensus.cc:686] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:45.184239 29316 consensus_queue.cc:237] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:46.104791 26046 raft_consensus.cc:3037] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:45.528365 25932 raft_consensus.cc:1226] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 5. Ops: [3.3-3.3]
I20260807 11:10:45.540510 26037 raft_consensus.cc:2454] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:10:46.106120 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1549) took 3673 ms. Trace:
I20260807 11:10:45.472751 25916 raft_consensus.cc:2454] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
W20260807 11:10:45.850097 25870 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.003s sys 0.021s
W20260807 11:10:45.727759 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 449685us
I20260807 11:10:46.111109 25786 raft_consensus.cc:1226] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6. Ops: []
I20260807 11:10:46.106786 25932 rpcz_store.cc:276] 0807 11:10:42.432315 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:45.198120 (+2765805us) service_pool.cc:227] Handling call
0807 11:10:45.528029 (+329909us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:46.105774 (+577745us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:46.106043 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:45.562997 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1654) took 3515 ms. Trace:
I20260807 11:10:46.161191 25916 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1685) took 1124 ms. Trace:
W20260807 11:10:46.161424 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 731292us
I20260807 11:10:46.163700 25916 rpcz_store.cc:276] 0807 11:10:45.037217 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:45.136347 (+ 99130us) service_pool.cc:227] Handling call
0807 11:10:46.161121 (+1024774us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:45.263311 29397 raft_consensus.cc:482] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 11:10:45.466496 29278 consensus_meta.cc:287] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.930s user 0.005s sys 0.006s
I20260807 11:10:45.160295 29334 raft_consensus.cc:3032] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:46.180204 26046 raft_consensus.cc:2396] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260807 11:10:46.161849 25870 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.894s user 0.011s sys 0.035s
I20260807 11:10:46.163106 25921 rpcz_store.cc:276] 0807 11:10:42.047118 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:45.139083 (+3091965us) service_pool.cc:227] Handling call
0807 11:10:45.562495 (+423412us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:45.562683 (+ 188us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:45.562708 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:45.562747 (+ 39us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:45.562777 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:45.562826 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:45.562940 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:46.165959 25752 leader_election.cc:302] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:46.108091 25869 leader_election.cc:302] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:45.522485 25796 raft_consensus.cc:2454] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:46.171657 25800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1528) took 1118 ms. Trace:
W20260807 11:10:45.510789 29511 log.cc:924] Time spent T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.351s user 0.000s sys 0.000s
I20260807 11:10:45.269954 29273 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
W20260807 11:10:46.084735 29179 consensus_meta.cc:287] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.913s user 0.009s sys 0.001s
W20260807 11:10:46.066321 29244 consensus_meta.cc:287] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 1.039s user 0.013s sys 0.000s
I20260807 11:10:46.182869 25800 rpcz_store.cc:276] 0807 11:10:45.060049 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:45.268189 (+208140us) service_pool.cc:227] Handling call
0807 11:10:46.171584 (+903395us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:10:46.226696 26034 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 1727) took 4587 ms (client timeout 1748 ms). Trace:
I20260807 11:10:46.256646 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
W20260807 11:10:46.241863 25750 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.670s user 0.002s sys 0.006s
W20260807 11:10:46.259101 25750 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.688s user 0.003s sys 0.006s
W20260807 11:10:46.232595 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 124591us
W20260807 11:10:46.259416 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 688222us
I20260807 11:10:45.495978 29267 raft_consensus.cc:2726] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:45.520373 29129 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:45.180721 29347 leader_election.cc:288] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:46.018880 29318 leader_election.cc:288] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:46.266562 29525 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:46.286068 29525 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:46.260857 29273 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:46.289193 29196 consensus_queue.cc:237] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:45.493708 29169 consensus_queue.cc:260] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:46.291780 29273 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:46.292146 29169 raft_consensus.cc:3037] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
W20260807 11:10:46.297547 29512 log.cc:924] Time spent T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 1.304s user 0.010s sys 0.031s
I20260807 11:10:46.262827 29315 raft_consensus.cc:482] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
W20260807 11:10:45.814639 29307 consensus_meta.cc:287] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 1.009s user 0.008s sys 0.003s
I20260807 11:10:46.302717 29307 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:45.406646 29523 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:46.037369 29198 consensus_meta.cc:287] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 1.037s user 0.009s sys 0.000s
I20260807 11:10:46.305047 26038 raft_consensus.cc:1226] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:46.305203 25803 raft_consensus.cc:2454] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:46.183468 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59312 (request call id 1656) took 1045 ms. Trace:
W20260807 11:10:46.257783 26034 rpcz_store.cc:269] 0807 11:10:41.589435 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:41.605642 (+ 16207us) service_pool.cc:227] Handling call
0807 11:10:46.168101 (+4562459us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b5400312754. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:46.177292 (+ 9191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2626922752}
I20260807 11:10:46.306085 26046 rpcz_store.cc:276] 0807 11:10:45.138085 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:45.264266 (+126181us) service_pool.cc:227] Handling call
0807 11:10:46.183401 (+919135us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:46.306769 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:46.307006 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:46.307132 26985 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 433) took 2271 ms (client timeout 1620 ms). Trace:
I20260807 11:10:46.307659 26046 raft_consensus.cc:2454] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
W20260807 11:10:46.307785 26985 rpcz_store.cc:269] 0807 11:10:44.036023 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.200650 (+164627us) service_pool.cc:227] Handling call
0807 11:10:46.306938 (+2106288us) spinlock_profiling.cc:243] Waited 514 ms on lock 0x7b54004c13d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:46.307052 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1181664512}
I20260807 11:10:46.308218 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:46.309450 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:46.242730 25808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53324 (request call id 1524) took 1190 ms. Trace:
W20260807 11:10:45.817715 29521 log.cc:924] Time spent T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.419s user 0.000s sys 0.000s
I20260807 11:10:46.329625 26938 leader_election.cc:302] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:46.331264 29084 raft_consensus.cc:2726] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 11:10:46.105930 29283 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:45.518139 29272 raft_consensus.cc:482] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:45.305584 28797 raft_consensus.cc:2726] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:46.165102 29397 raft_consensus.cc:3037] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:45.512060 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:46.142210 29515 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:46.183853 29352 consensus_meta.cc:287] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 1.281s user 0.008s sys 0.004s
W20260807 11:10:46.184839 29295 consensus_meta.cc:287] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 1.232s user 0.012s sys 0.000s
I20260807 11:10:46.269017 25921 raft_consensus.cc:1226] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:46.140808 29520 raft_consensus.cc:482] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:46.172739 29013 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:46.349939 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 241) took 3737 ms. Trace:
I20260807 11:10:46.350507 25921 rpcz_store.cc:276] 0807 11:10:42.612907 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.182217 (+3569310us) service_pool.cc:227] Handling call
0807 11:10:46.268633 (+ 86416us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:46.349589 (+ 80956us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:46.349856 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:45.925209 29490 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:45.723754 29098 raft_consensus.cc:482] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:46.109258 29537 raft_consensus.cc:482] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:46.260562 29277 raft_consensus.cc:2726] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260807 11:10:46.184103 29205 consensus_meta.cc:287] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 1.344s user 0.010s sys 0.002s
I20260807 11:10:46.183650 29006 raft_consensus.cc:2726] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 11:10:45.300815 29253 raft_consensus.cc:504] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:45.933331 29223 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:46.304817 29523 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:46.308691 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:10:45.722988 29029 consensus_meta.cc:287] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 1.166s user 0.009s sys 0.002s
W20260807 11:10:46.115872 29505 consensus_meta.cc:287] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 1.115s user 0.011s sys 0.000s
W20260807 11:10:46.309998 26035 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 1734) took 4718 ms (client timeout 2114 ms). Trace:
I20260807 11:10:45.561604 29440 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:46.248050 29373 consensus_meta.cc:287] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 1.241s user 0.008s sys 0.004s
I20260807 11:10:46.309857 29307 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:46.267344 28647 raft_consensus.cc:482] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:46.325997 29315 raft_consensus.cc:504] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:46.172739 29500 consensus_queue.cc:260] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:46.414760 29500 raft_consensus.cc:3037] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:46.415160 28647 raft_consensus.cc:504] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:46.415831 29315 leader_election.cc:288] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:46.411939 29205 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:46.417467 28647 leader_election.cc:288] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:46.369575 29098 raft_consensus.cc:504] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:46.178485 29254 raft_consensus.cc:504] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:46.327064 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:45.763866 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:46.420192 26974 raft_consensus.cc:2454] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:46.260897 29179 raft_consensus.cc:2726] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:46.328358 29525 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:46.420944 29098 leader_election.cc:288] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:45.725932 28946 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:46.422302 29523 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:46.178582 29334 raft_consensus.cc:729] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:46.344293 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:10:46.420861 26974 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 437) took 1375 ms (client timeout 1667 ms). Trace:
I20260807 11:10:46.424278 28946 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:46.426695 26985 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 410) took 1390 ms (client timeout 1785 ms). Trace:
W20260807 11:10:46.426594 26974 rpcz_store.cc:269] 0807 11:10:45.045527 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:45.496752 (+451225us) service_pool.cc:227] Handling call
0807 11:10:46.420786 (+924034us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:10:46.427227 26985 rpcz_store.cc:269] 0807 11:10:45.035983 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.310529 (+1274546us) service_pool.cc:227] Handling call
0807 11:10:46.426616 (+116087us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:45.672326 29454 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:46.375380 29441 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:46.428424 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:46.429296 26985 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 440) took 1234 ms. Trace:
I20260807 11:10:46.428874 29441 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:46.429651 26985 rpcz_store.cc:276] 0807 11:10:45.194726 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.428033 (+1233307us) service_pool.cc:227] Handling call
0807 11:10:46.429238 (+ 1205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":51968}
I20260807 11:10:46.370579 29537 raft_consensus.cc:504] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:46.346809 29111 raft_consensus.cc:3032] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:46.347370 29515 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:46.366232 29013 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:46.430893 29111 raft_consensus.cc:729] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:46.431888 29111 consensus_queue.cc:260] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:46.432706 29515 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:46.433135 29111 raft_consensus.cc:3037] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:46.434965 26978 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 434) took 1963 ms. Trace:
I20260807 11:10:46.435298 26978 rpcz_store.cc:276] 0807 11:10:44.471848 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.783882 (+312034us) service_pool.cc:227] Handling call
0807 11:10:46.434299 (+1650417us) spinlock_profiling.cc:243] Waited 8.16 ms on lock 0x7b5800000f98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc461a2b0 00007f3fc46057f9 00007f3fc46052a2 00007f3fbdceb684
0807 11:10:46.434486 (+ 187us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:46.434675 (+ 189us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 11:10:46.434708 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:46.434739 (+ 31us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:46.434776 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:46.434820 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:46.434906 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18770304}
I20260807 11:10:46.417497 29205 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:46.412997 29373 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:46.438838 26984 raft_consensus.cc:2454] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:46.439184 29373 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:46.439435 29205 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:46.365142 29295 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:46.442113 29537 leader_election.cc:288] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:46.411984 29440 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:46.443930 26980 raft_consensus.cc:1226] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 2. Current term is 4. Ops: []
I20260807 11:10:46.444005 29334 consensus_queue.cc:260] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:46.445192 29440 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:46.445376 29334 raft_consensus.cc:3037] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:46.410923 29505 raft_consensus.cc:2726] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:46.410820 29029 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:46.447095 29544 log.cc:924] Time spent T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.337s user 0.001s sys 0.000s
I20260807 11:10:46.447855 29029 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:46.343755 29272 raft_consensus.cc:504] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:46.349097 29352 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:46.366460 29520 raft_consensus.cc:504] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:46.342832 29283 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:46.381213 26975 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 438) took 1316 ms (client timeout 1685 ms). Trace:
I20260807 11:10:46.425137 29253 leader_election.cc:288] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:46.442595 29295 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:10:46.413988 29524 log.cc:924] Time spent T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.149s user 0.000s sys 0.001s
I20260807 11:10:46.454442 29352 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:46.451895 29272 leader_election.cc:288] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:46.449843 26984 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50960 (request call id 339) took 4737 ms (client timeout 2126 ms). Trace:
I20260807 11:10:46.455062 26987 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50942 (request call id 408) took 1447 ms. Trace:
I20260807 11:10:46.455618 29520 leader_election.cc:288] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:46.425850 29254 leader_election.cc:288] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:46.452741 26975 rpcz_store.cc:269] 0807 11:10:45.055366 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:45.511772 (+456406us) service_pool.cc:227] Handling call
0807 11:10:46.371405 (+859633us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:46.450523 29397 raft_consensus.cc:504] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:46.456179 26987 rpcz_store.cc:276] 0807 11:10:45.007161 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:45.007325 (+ 164us) service_pool.cc:227] Handling call
0807 11:10:45.007716 (+ 391us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:10:45.008169 (+ 453us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:45.008201 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 11:10:45.008747 (+ 546us) log.cc:842] Serialized 52 byte log entry
0807 11:10:45.405157 (+396410us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:45.405197 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:45.405267 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:46.454837 (+1049570us) raft_consensus.cc:1671] finished
0807 11:10:46.454860 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:46.454996 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4913,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":85,"thread_start_us":396010,"threads_started":1,"wal-append.queue_time_us":937199}
I20260807 11:10:46.461472 29454 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:46.461651 29397 leader_election.cc:288] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:46.457705 26984 rpcz_store.cc:269] 0807 11:10:41.705479 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:41.705612 (+ 133us) service_pool.cc:227] Handling call
0807 11:10:46.440726 (+4735114us) spinlock_profiling.cc:243] Waited 4.69 s on lock 0x7b5400710554. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:46.443249 (+ 2523us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10796280576}
I20260807 11:10:46.463804 29330 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:46.459721 29510 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.471s user 0.020s sys 0.026s
W20260807 11:10:46.465219 29510 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.479s user 0.020s sys 0.026s
I20260807 11:10:46.466507 29191 consensus_queue.cc:1046] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:10:46.467365 29191 consensus_queue.cc:1225] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer b22d69db45194908ad3d345eba9badd9 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260807 11:10:46.471155 29250 raft_consensus.cc:3032] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:46.471563 29250 raft_consensus.cc:729] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:46.472414 29250 consensus_queue.cc:260] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:46.473281 29509 raft_consensus.cc:3032] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:46.473748 29250 raft_consensus.cc:3037] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:46.473795 29509 raft_consensus.cc:729] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:46.474906 29509 consensus_queue.cc:260] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:46.476501 29509 raft_consensus.cc:3037] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:46.300998 25797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 305) took 3424 ms. Trace:
I20260807 11:10:46.520913 25797 rpcz_store.cc:276] 0807 11:10:42.877056 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:43.926662 (+1049606us) service_pool.cc:227] Handling call
0807 11:10:43.944565 (+ 17903us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:44.921042 (+976477us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:46.300699 (+1379657us) spinlock_profiling.cc:243] Waited 975 ms on lock 0x7b5400266e54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:46.300932 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2243248896}
I20260807 11:10:46.182624 25796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1672) took 1156 ms. Trace:
I20260807 11:10:46.527462 25786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53324 (request call id 1516) took 2912 ms. Trace:
I20260807 11:10:46.527380 25796 rpcz_store.cc:276] 0807 11:10:45.026382 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:45.026792 (+ 410us) service_pool.cc:227] Handling call
0807 11:10:46.182547 (+1155755us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:46.529564 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:46.530471 25799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 301) took 3803 ms. Trace:
I20260807 11:10:46.524057 25750 leader_election.cc:302] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:46.531030 25799 rpcz_store.cc:276] 0807 11:10:42.726927 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:43.881868 (+1154941us) service_pool.cc:227] Handling call
0807 11:10:43.894487 (+ 12619us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:44.553002 (+658515us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:46.529603 (+1976601us) spinlock_profiling.cc:243] Waited 640 ms on lock 0x7b5400265a54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:46.530150 (+ 547us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1472002304}
I20260807 11:10:46.532361 25801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 306) took 3645 ms. Trace:
I20260807 11:10:46.310765 25808 rpcz_store.cc:276] 0807 11:10:45.052243 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:45.053848 (+ 1605us) service_pool.cc:227] Handling call
0807 11:10:45.529395 (+475547us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:45.529678 (+ 283us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:45.529721 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:45.573188 (+ 43467us) log.cc:842] Serialized 26 byte log entry
0807 11:10:46.242228 (+669040us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:10:46.242291 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:46.242392 (+ 101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:46.242456 (+ 64us) raft_consensus.cc:1671] finished
0807 11:10:46.242476 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:46.242646 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":406470,"threads_started":1,"wal-append.queue_time_us":419461}
I20260807 11:10:46.533181 25801 rpcz_store.cc:276] 0807 11:10:42.887116 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.519520 (+1632404us) service_pool.cc:227] Handling call
0807 11:10:44.519826 (+ 306us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:45.191742 (+671916us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:46.531733 (+1339991us) spinlock_profiling.cc:243] Waited 671 ms on lock 0x7b54002f6a54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:46.532159 (+ 426us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1543712128}
I20260807 11:10:46.533977 25800 raft_consensus.cc:2379] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 5: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:46.534682 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:46.535076 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:46.535912 25808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1539) took 1068 ms. Trace:
I20260807 11:10:46.536111 25801 raft_consensus.cc:2454] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
W20260807 11:10:46.263971 25798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1523) took 2444 ms (client timeout 1505 ms). Trace:
I20260807 11:10:46.536378 25808 rpcz_store.cc:276] 0807 11:10:45.468031 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.533989 (+1065958us) service_pool.cc:227] Handling call
0807 11:10:46.535813 (+ 1824us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:10:46.537561 25798 rpcz_store.cc:269] 0807 11:10:43.819652 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.263753 (+2444101us) service_pool.cc:214] Skipping call since client already timed out
0807 11:10:46.263879 (+ 126us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260807 11:10:46.538993 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:46.539506 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:46.540145 25808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1527) took 1487 ms. Trace:
I20260807 11:10:46.540632 25798 raft_consensus.cc:3037] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 1 FOLLOWER]: Advancing to term 2
W20260807 11:10:46.542126 25800 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1534) took 1435 ms (client timeout 1820 ms). Trace:
I20260807 11:10:46.528152 25786 rpcz_store.cc:276] 0807 11:10:43.614431 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.643187 (+1028756us) service_pool.cc:227] Handling call
0807 11:10:44.697703 (+ 54516us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:46.162682 (+1464979us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:46.526555 (+363873us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b5400136854. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:46.527125 (+ 570us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3249901952}
I20260807 11:10:46.530787 25796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1531) took 1469 ms. Trace:
W20260807 11:10:46.542842 25800 rpcz_store.cc:269] 0807 11:10:45.106508 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.256344 (+1149836us) service_pool.cc:227] Handling call
0807 11:10:46.541774 (+285430us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b5400265554. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:46.542017 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":623905152}
I20260807 11:10:46.541319 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:46.543303 25796 rpcz_store.cc:276] 0807 11:10:45.061023 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.528872 (+1467849us) service_pool.cc:227] Handling call
0807 11:10:46.530692 (+ 1820us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:46.550181 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:46.558996 25750 leader_election.cc:302] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:46.559304 25796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1541) took 1065 ms. Trace:
I20260807 11:10:46.561203 25796 rpcz_store.cc:276] 0807 11:10:45.494214 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.549563 (+1055349us) service_pool.cc:227] Handling call
0807 11:10:46.559223 (+ 9660us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:46.545774 25803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1669) took 1538 ms. Trace:
I20260807 11:10:46.543910 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:42.575178 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:10:46.569198 25803 rpcz_store.cc:276] 0807 11:10:45.007002 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:45.007142 (+ 140us) service_pool.cc:227] Handling call
0807 11:10:46.545329 (+1538187us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54002f5b54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:46.545531 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2979114240}
W20260807 11:10:46.569866 25919 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1671) took 2657 ms (client timeout 1523 ms). Trace:
I20260807 11:10:46.570076 25796 raft_consensus.cc:1226] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5. Ops: []
W20260807 11:10:46.570415 25919 rpcz_store.cc:269] 0807 11:10:43.912489 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:43.917093 (+ 4604us) service_pool.cc:227] Handling call
0807 11:10:46.569258 (+2652165us) spinlock_profiling.cc:243] Waited 981 ms on lock 0x7b54000352d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:46.569569 (+ 311us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2256379648}
I20260807 11:10:46.571017 25796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53324 (request call id 1525) took 1518 ms. Trace:
I20260807 11:10:46.571306 25803 raft_consensus.cc:1226] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6. Ops: []
I20260807 11:10:46.571466 25796 rpcz_store.cc:276] 0807 11:10:45.052574 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.569345 (+1516771us) service_pool.cc:227] Handling call
0807 11:10:46.569846 (+ 501us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:46.570674 (+ 828us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:46.570919 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:46.572157 25803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53324 (request call id 1526) took 1519 ms. Trace:
I20260807 11:10:46.573513 25918 raft_consensus.cc:1226] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:46.572592 25803 rpcz_store.cc:276] 0807 11:10:45.052867 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.570448 (+1517581us) service_pool.cc:227] Handling call
0807 11:10:46.571018 (+ 570us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:46.571912 (+ 894us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:46.572101 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:46.576423 25918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1551) took 4143 ms. Trace:
W20260807 11:10:46.553542 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 293455us
I20260807 11:10:46.544914 25799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1532) took 1483 ms. Trace:
I20260807 11:10:46.540637 25808 rpcz_store.cc:276] 0807 11:10:45.053169 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.538413 (+1485244us) service_pool.cc:227] Handling call
0807 11:10:46.540078 (+ 1665us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:46.525398 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:46.555721 25654 catalog_manager.cc:5495] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 1 to 2, leader changed from 4fd6381544574ddca559ce63b9243efc (127.23.120.130) to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 2 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } }
I20260807 11:10:46.545140 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:46.566030 25875 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.909s user 0.020s sys 0.051s
I20260807 11:10:46.567792 25916 raft_consensus.cc:1226] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260807 11:10:46.551185 25801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1530) took 1476 ms. Trace:
I20260807 11:10:46.577142 25786 raft_consensus.cc:2454] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:46.584941 25801 rpcz_store.cc:276] 0807 11:10:45.060700 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.534619 (+1473919us) service_pool.cc:227] Handling call
0807 11:10:46.536714 (+ 2095us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:10:46.345419 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 240317us
W20260807 11:10:46.393565 26035 rpcz_store.cc:269] 0807 11:10:41.591569 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:41.643393 (+ 51824us) service_pool.cc:227] Handling call
0807 11:10:46.309779 (+4666386us) spinlock_profiling.cc:243] Waited 989 ms on lock 0x7b5400229a54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:46.309919 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2274159360}
I20260807 11:10:46.371346 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:46.428452 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
W20260807 11:10:46.588137 26974 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 439) took 1393 ms (client timeout 1782 ms). Trace:
I20260807 11:10:46.588353 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:46.588737 26974 rpcz_store.cc:269] 0807 11:10:45.194420 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.428007 (+1233587us) service_pool.cc:227] Handling call
0807 11:10:46.587695 (+159688us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":32512}
I20260807 11:10:46.589186 29520 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:46.572342 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:46.589883 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 394) took 2610 ms. Trace:
I20260807 11:10:46.590718 25919 raft_consensus.cc:3037] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:46.590294 29520 raft_consensus.cc:504] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:46.578452 25799 rpcz_store.cc:276] 0807 11:10:45.061329 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.540821 (+1479492us) service_pool.cc:227] Handling call
0807 11:10:46.544833 (+ 4012us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:10:46.577555 25917 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1528) took 4949 ms (client timeout 1824 ms). Trace:
I20260807 11:10:46.603312 25786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1536) took 1443 ms. Trace:
I20260807 11:10:46.605031 26041 raft_consensus.cc:1226] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260807 11:10:46.605513 29410 raft_consensus.cc:482] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:46.606475 25869 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:46.606784 29191 consensus_queue.cc:1035] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260807 11:10:46.608170 25875 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.218s user 0.021s sys 0.052s
I20260807 11:10:46.609290 25796 raft_consensus.cc:1261] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 11:10:46.609447 29334 raft_consensus.cc:482] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:46.610318 29407 raft_consensus.cc:482] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:46.611340 29510 raft_consensus.cc:2781] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:10:46.613328 25654 catalog_manager.cc:5495] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:46.590449 26046 rpcz_store.cc:276] 0807 11:10:43.978832 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.587980 (+2609148us) service_pool.cc:227] Handling call
0807 11:10:46.588696 (+ 716us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:46.589251 (+ 555us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:46.589302 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:46.589367 (+ 65us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:46.589409 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:46.589463 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:46.589579 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:46.578142 25918 rpcz_store.cc:276] 0807 11:10:42.432901 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.165359 (+3732458us) service_pool.cc:227] Handling call
0807 11:10:46.572926 (+407567us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:46.573987 (+ 1061us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:46.576158 (+ 2171us) spinlock_profiling.cc:243] Waited 407 ms on lock 0x7b54000352d0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:46.576318 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":936131328}
I20260807 11:10:46.618837 25786 rpcz_store.cc:276] 0807 11:10:45.142658 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.543459 (+1400801us) service_pool.cc:227] Handling call
0807 11:10:46.585812 (+ 42353us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:46.622289 26034 raft_consensus.cc:2379] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 6: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:46.622095 29352 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:46.623258 26039 raft_consensus.cc:1226] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 2. Current term is 3. Ops: []
W20260807 11:10:46.623620 26033 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 1724) took 5035 ms (client timeout 1557 ms). Trace:
I20260807 11:10:46.625588 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1748) took 3845 ms. Trace:
I20260807 11:10:46.626314 26975 raft_consensus.cc:3032] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:46.627224 25921 raft_consensus.cc:1226] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260807 11:10:46.776427 29505 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:46.777237 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 242) took 4163 ms. Trace:
I20260807 11:10:46.777307 29505 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:46.777740 25921 rpcz_store.cc:276] 0807 11:10:42.613625 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.353279 (+3739654us) service_pool.cc:227] Handling call
0807 11:10:46.626615 (+273336us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:46.776873 (+150258us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:46.777161 (+ 288us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:46.779098 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:46.627782 25800 raft_consensus.cc:2454] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:46.780823 25800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1538) took 1609 ms. Trace:
I20260807 11:10:46.780934 25808 raft_consensus.cc:1261] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 11:10:46.776180 29352 raft_consensus.cc:504] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:46.781219 25800 rpcz_store.cc:276] 0807 11:10:45.171208 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.544713 (+1373505us) service_pool.cc:227] Handling call
0807 11:10:46.780716 (+236003us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:46.627867 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:46.629321 29520 leader_election.cc:288] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:46.629412 26974 raft_consensus.cc:3032] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 11:10:46.630159 29440 raft_consensus.cc:482] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:46.631341 26984 raft_consensus.cc:1226] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260807 11:10:46.632890 25797 raft_consensus.cc:2454] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 5.
I20260807 11:10:46.633957 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:46.635143 29205 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
W20260807 11:10:46.602802 25917 rpcz_store.cc:269] 0807 11:10:41.628042 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:41.791733 (+163691us) service_pool.cc:227] Handling call
0807 11:10:46.577136 (+4785403us) spinlock_profiling.cc:243] Waited 2.72 s on lock 0x7b54001a09d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:46.577346 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6258768256}
I20260807 11:10:46.638828 29515 raft_consensus.cc:482] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
W20260807 11:10:46.652709 25931 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1564) took 3028 ms (client timeout 1988 ms). Trace:
I20260807 11:10:46.639736 25981 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:46.641191 29330 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:46.654222 26037 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 1754) took 2760 ms (client timeout 2490 ms). Trace:
I20260807 11:10:46.641429 29307 raft_consensus.cc:482] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:46.642166 29254 raft_consensus.cc:2781] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:10:46.640504 25796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53324 (request call id 1533) took 1578 ms. Trace:
I20260807 11:10:46.643868 29272 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:46.655819 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 385) took 3838 ms. Trace:
I20260807 11:10:46.773051 25798 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
I20260807 11:10:46.772449 26041 rpcz_store.cc:276] 0807 11:10:42.780315 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.545354 (+1765039us) service_pool.cc:227] Handling call
0807 11:10:44.559396 (+ 14042us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:46.623352 (+2063956us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:46.625167 (+ 1815us) spinlock_profiling.cc:243] Waited 2.04 s on lock 0x7b540022bd54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:46.625460 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4695516928}
I20260807 11:10:46.774435 26975 raft_consensus.cc:729] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:46.780022 25921 raft_consensus.cc:2379] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 5: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
W20260807 11:10:46.636107 25660 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34692 (request call id 134) took 20958 ms (client timeout 15000 ms). Trace:
W20260807 11:10:46.632947 26036 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43246 (request call id 390) took 3351 ms (client timeout 1996 ms). Trace:
W20260807 11:10:46.665318 26986 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 399) took 2683 ms (client timeout 1758 ms). Trace:
W20260807 11:10:46.772444 26033 rpcz_store.cc:269] 0807 11:10:41.588329 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:41.593280 (+ 4951us) service_pool.cc:227] Handling call
0807 11:10:46.623094 (+5029814us) spinlock_profiling.cc:243] Waited 3.05 s on lock 0x7b54000a3f54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:46.623379 (+ 285us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7015186688}
I20260807 11:10:46.642907 25869 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:46.664026 26979 raft_consensus.cc:1226] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:46.645908 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:46.662341 26980 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50942 (request call id 391) took 4110 ms. Trace:
I20260807 11:10:46.649250 29415 raft_consensus.cc:2781] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260807 11:10:47.039369 25798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1529) took 1979 ms (client timeout 2224 ms). Trace:
W20260807 11:10:47.102650 25798 rpcz_store.cc:269] 0807 11:10:45.060391 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.538975 (+1478584us) service_pool.cc:227] Handling call
0807 11:10:47.039300 (+500325us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.043468 29516 raft_consensus.cc:2781] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260807 11:10:47.047307 29250 consensus_meta.cc:287] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.573s user 0.013s sys 0.002s
I20260807 11:10:47.049293 29272 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:47.050637 29307 raft_consensus.cc:504] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:47.041594 26037 rpcz_store.cc:269] 0807 11:10:43.893882 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.399145 (+505263us) service_pool.cc:227] Handling call
0807 11:10:46.653687 (+2254542us) spinlock_profiling.cc:243] Waited 675 ms on lock 0x7b54000a4e54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:46.653943 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1552081152}
I20260807 11:10:47.059368 25750 leader_election.cc:302] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:47.060132 29558 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:47.041468 26980 rpcz_store.cc:276] 0807 11:10:42.551686 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:42.578450 (+ 26764us) service_pool.cc:227] Handling call
0807 11:10:46.443702 (+3865252us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:46.444578 (+ 876us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:46.661742 (+217164us) spinlock_profiling.cc:243] Waited 3.85 s on lock 0x7b5400710550. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:46.662046 (+ 304us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8852047232}
W20260807 11:10:47.036736 26036 rpcz_store.cc:269] 0807 11:10:43.281352 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:43.294756 (+ 13404us) service_pool.cc:227] Handling call
0807 11:10:46.632363 (+3337607us) spinlock_profiling.cc:243] Waited 622 ms on lock 0x7b54000bbb54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:46.632621 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1431492864}
I20260807 11:10:46.787155 26974 raft_consensus.cc:729] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:47.037039 25796 rpcz_store.cc:276] 0807 11:10:45.061620 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.572396 (+1510776us) service_pool.cc:227] Handling call
0807 11:10:46.601601 (+ 29205us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:46.640078 (+ 38477us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:46.640401 (+ 323us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.054178 29445 raft_consensus.cc:3032] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:46.638443 29373 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:47.039021 26986 rpcz_store.cc:269] 0807 11:10:43.981309 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:43.983291 (+ 1982us) service_pool.cc:227] Handling call
0807 11:10:46.664713 (+2681422us) spinlock_profiling.cc:243] Waited 407 ms on lock 0x7b540069e954. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:46.665054 (+ 341us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":936824064}
W20260807 11:10:47.037336 25797 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1535) took 1895 ms (client timeout 1809 ms). Trace:
I20260807 11:10:47.040747 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:47.036624 25931 rpcz_store.cc:269] 0807 11:10:43.624101 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.041306 (+417205us) service_pool.cc:227] Handling call
0807 11:10:46.617947 (+2576641us) spinlock_profiling.cc:243] Waited 805 ms on lock 0x7b5400054954. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:46.652457 (+ 34510us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1851167488}
I20260807 11:10:46.774391 29510 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:47.038689 26042 rpcz_store.cc:276] 0807 11:10:42.817598 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.806236 (+1988638us) service_pool.cc:227] Handling call
0807 11:10:44.806529 (+ 293us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:45.177413 (+370884us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:46.655258 (+1477845us) spinlock_profiling.cc:243] Waited 370 ms on lock 0x7b540038d954. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:46.655628 (+ 370us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":851484544}
I20260807 11:10:47.030717 29352 leader_election.cc:288] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:46.637342 29410 raft_consensus.cc:504] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:47.123778 25797 rpcz_store.cc:269] 0807 11:10:45.142275 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.524476 (+1382201us) service_pool.cc:227] Handling call
0807 11:10:47.037256 (+512780us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:46.814448 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:47.061023 29179 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:47.053385 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:47.128790 26985 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 1.
W20260807 11:10:47.132467 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 73324.7us
I20260807 11:10:47.114434 29505 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:47.117080 26979 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50960 (request call id 346) took 3497 ms. Trace:
I20260807 11:10:46.794948 29537 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:47.135696 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:47.138509 29179 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:47.140074 29440 raft_consensus.cc:504] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:47.140970 29558 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:47.142187 29191 raft_consensus.cc:2781] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:10:47.190876 29283 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:47.190974 29191 raft_consensus.cc:686] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:47.191955 29261 raft_consensus.cc:2781] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:47.149885 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:47.192436 29261 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:47.192991 29261 raft_consensus.cc:3037] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:47.193521 29179 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:47.193691 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:47.194144 29179 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:47.194691 26985 raft_consensus.cc:2354] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6.
I20260807 11:10:47.194881 29179 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:47.188081 29123 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:47.195837 29441 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:47.197113 29479 raft_consensus.cc:3032] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Stepping down as leader of term 1
I20260807 11:10:47.197692 29479 raft_consensus.cc:729] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:47.198122 29440 leader_election.cc:288] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:47.199846 29558 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:47.199247 29479 consensus_queue.cc:260] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:47.153836 29352 raft_consensus.cc:2741] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260807 11:10:47.154639 25803 raft_consensus.cc:1261] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 11:10:47.151940 29013 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:47.200946 25803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53324 (request call id 1537) took 2046 ms. Trace:
I20260807 11:10:47.151677 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:46.860008 29334 raft_consensus.cc:504] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:47.157086 25918 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:47.168939 29330 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:47.170851 25798 raft_consensus.cc:1261] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 11:10:47.171203 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:47.171487 29560 log.cc:924] Time spent T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.376s user 0.000s sys 0.002s
I20260807 11:10:47.163746 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1760) took 3228 ms. Trace:
I20260807 11:10:47.171661 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:47.172499 25786 raft_consensus.cc:1226] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 5. Current term is 6. Ops: []
I20260807 11:10:47.164572 25808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53324 (request call id 1540) took 1674 ms. Trace:
I20260807 11:10:47.176928 25797 raft_consensus.cc:1261] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260807 11:10:47.177518 26975 consensus_queue.cc:260] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:47.166844 29410 leader_election.cc:288] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:47.164860 25752 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.662s user 0.028s sys 0.018s
I20260807 11:10:47.176931 29515 raft_consensus.cc:504] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:47.179116 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:47.036676 25660 rpcz_store.cc:269] 0807 11:10:25.677066 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:25.677639 (+ 573us) service_pool.cc:227] Handling call
0807 11:10:46.635642 (+20958003us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 11:10:36.806807 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 11:10:36.807412 (+ 605us) write_op.cc:432] Acquiring schema lock in shared mode
0807 11:10:36.807465 (+ 53us) write_op.cc:435] Acquired schema lock
0807 11:10:36.807481 (+ 16us) tablet.cc:660] Decoding operations
0807 11:10:36.808655 (+ 1174us) write_op.cc:620] Acquiring the partition lock for write op
0807 11:10:36.814332 (+ 5677us) write_op.cc:641] Partition lock acquired for write op
0807 11:10:36.814361 (+ 29us) tablet.cc:683] Acquiring locks for 12 operations
0807 11:10:36.815550 (+ 1189us) tablet.cc:699] Row locks acquired
0807 11:10:36.815595 (+ 45us) write_op.cc:260] PREPARE: finished
0807 11:10:36.815786 (+ 191us) write_op.cc:270] Start()
0807 11:10:36.815961 (+ 175us) write_op.cc:276] Timestamp: P: 1786101036815707 usec, L: 0
0807 11:10:36.815998 (+ 37us) op_driver.cc:331] REPLICATION: starting
0807 11:10:36.816732 (+ 734us) log.cc:842] Serialized 4978 byte log entry
0807 11:10:37.645412 (+828680us) spinlock_profiling.cc:243] Waited 7.62 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:10:38.113002 (+467590us) op_driver.cc:444] REPLICATION: finished
0807 11:10:38.181597 (+ 68595us) write_op.cc:301] APPLY: starting
0807 11:10:38.181737 (+ 140us) tablet.cc:1365] starting BulkCheckPresence
0807 11:10:38.182136 (+ 399us) tablet.cc:1368] finished BulkCheckPresence
0807 11:10:38.183418 (+ 1282us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 11:10:38.218628 (+ 35210us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 11:10:38.218977 (+ 349us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=12
0807 11:10:38.219031 (+ 54us) write_op.cc:312] APPLY: finished
0807 11:10:38.219487 (+ 456us) log.cc:842] Serialized 89 byte log entry
0807 11:10:38.219708 (+ 221us) write_op.cc:489] Releasing partition, row and schema locks
0807 11:10:38.219877 (+ 169us) write_op.cc:454] Released schema lock
0807 11:10:38.219952 (+ 75us) write_op.cc:341] FINISH: Updating metrics
0807 11:10:38.298243 (+ 78291us) write_op.cc:489] Releasing partition, row and schema locks
0807 11:10:38.298852 (+ 609us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1640008,"reported_tablets":25,"child_traces":[["op",{"apply.queue_time_us":68151,"apply.run_cpu_time_us":4994,"apply.run_wall_time_us":145952,"num_ops":12,"prepare.queue_time_us":240073,"prepare.run_cpu_time_us":40418,"prepare.run_wall_time_us":484881,"raft.queue_time_us":347070,"raft.run_cpu_time_us":7090,"raft.run_wall_time_us":90614,"replication_time_us":1296867,"spinlock_wait_cycles":17532288,"thread_start_us":886257,"threads_started":4,"wal-append.queue_time_us":580691,"wal-append.run_cpu_time_us":52275,"wal-append.run_wall_time_us":1862220}]]}
I20260807 11:10:47.181002 29307 leader_election.cc:288] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:47.176429 28438 leader_election.cc:288] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:47.181149 26033 raft_consensus.cc:2454] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:47.123224 29407 raft_consensus.cc:504] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:47.155133 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 515776us
I20260807 11:10:47.137807 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:47.182530 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:47.182699 29272 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:47.158555 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 391) took 3746 ms. Trace:
I20260807 11:10:47.197221 29001 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:47.223191 26038 rpcz_store.cc:276] 0807 11:10:43.411724 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.952612 (+1540888us) service_pool.cc:227] Handling call
0807 11:10:44.952932 (+ 320us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:46.305766 (+1352834us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.157868 (+852102us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b540029ed54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:47.158270 (+ 402us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3108933888}
I20260807 11:10:47.223976 26033 raft_consensus.cc:3032] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 11:10:47.223572 29001 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:47.224355 26033 raft_consensus.cc:729] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:47.224602 25806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1665) took 2670 ms. Trace:
I20260807 11:10:47.225337 26038 raft_consensus.cc:1261] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 11:10:47.225564 26033 consensus_queue.cc:260] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:47.153271 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:47.226255 29001 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:47.226698 26035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 395) took 3191 ms. Trace:
I20260807 11:10:47.199671 29267 consensus_queue.cc:1046] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 11:10:47.227487 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:47.227967 29267 raft_consensus.cc:3032] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:47.228402 29267 raft_consensus.cc:729] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:47.129236 29510 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:47.228688 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:47.228937 29334 leader_election.cc:288] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:47.229630 26046 raft_consensus.cc:2454] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:47.229688 29267 consensus_queue.cc:260] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:47.231396 29267 raft_consensus.cc:3037] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:47.201287 25803 rpcz_store.cc:276] 0807 11:10:45.154923 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.575456 (+1420533us) service_pool.cc:227] Handling call
0807 11:10:47.153963 (+578507us) spinlock_profiling.cc:243] Waited 2.23 ms on lock 0x7b5800003c98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc461a2b0 00007f3fc46057f9 00007f3fc46052a2 00007f3fbdceb684
0807 11:10:47.154209 (+ 246us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.200610 (+ 46401us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.200877 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5122944}
I20260807 11:10:47.204324 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:47.201978 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:47.207614 25801 raft_consensus.cc:2454] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:10:47.121963 29445 raft_consensus.cc:729] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:47.203320 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:47.211302 25919 consensus_meta.cc:287] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.617s user 0.007s sys 0.014s
I20260807 11:10:47.206506 26039 rpcz_store.cc:276] 0807 11:10:43.935178 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:45.511772 (+1576594us) service_pool.cc:227] Handling call
0807 11:10:45.512104 (+ 332us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:46.776939 (+1264835us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.163417 (+386478us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b5400229f54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:47.163667 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2504440064}
I20260807 11:10:47.234309 25919 raft_consensus.cc:2454] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:47.213626 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
W20260807 11:10:47.216701 29246 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.093s user 0.008s sys 0.010s
I20260807 11:10:47.216590 29537 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:47.221728 25752 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.963s user 0.028s sys 0.018s
I20260807 11:10:47.224154 26036 raft_consensus.cc:1226] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 5 LEADER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:47.226050 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1647) took 3085 ms. Trace:
I20260807 11:10:47.207172 25808 rpcz_store.cc:276] 0807 11:10:45.490490 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.601581 (+1111091us) service_pool.cc:227] Handling call
0807 11:10:46.604735 (+ 3154us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:46.781318 (+176583us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.164294 (+382976us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b54000e38d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:47.164487 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":353355648}
I20260807 11:10:47.210345 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:47.238027 26038 rpcz_store.cc:276] 0807 11:10:44.140549 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.224360 (+3083811us) service_pool.cc:227] Handling call
0807 11:10:47.224738 (+ 378us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.225792 (+ 1054us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.225999 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.213197 25917 raft_consensus.cc:2396] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260807 11:10:47.246378 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:47.227073 26035 rpcz_store.cc:276] 0807 11:10:44.035322 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.589827 (+2554505us) service_pool.cc:227] Handling call
0807 11:10:46.590448 (+ 621us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:46.591093 (+ 645us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:46.591149 (+ 56us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:46.592215 (+ 1066us) log.cc:842] Serialized 26 byte log entry
0807 11:10:47.031817 (+439602us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:47.031962 (+ 145us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.032161 (+ 199us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:47.191309 (+159148us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:10:47.219170 (+ 27861us) raft_consensus.cc:1671] finished
0807 11:10:47.219198 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:47.226592 (+ 7394us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":28767616,"thread_start_us":190802,"threads_started":1,"wal-append.queue_time_us":562897}
I20260807 11:10:47.247886 25796 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:47.186889 26979 rpcz_store.cc:276] 0807 11:10:43.616518 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.662697 (+3046179us) service_pool.cc:227] Handling call
0807 11:10:46.663393 (+ 696us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.078316 (+414923us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.114046 (+ 35730us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.240309 25921 raft_consensus.cc:2454] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:47.245143 26036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1772) took 3127 ms. Trace:
I20260807 11:10:47.251101 29179 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:47.244181 29246 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.125s user 0.008s sys 0.011s
I20260807 11:10:47.225108 25806 rpcz_store.cc:276] 0807 11:10:44.551655 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.833346 (+281691us) service_pool.cc:227] Handling call
0807 11:10:44.853507 (+ 20161us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:44.853894 (+ 387us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:44.853928 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:46.527605 (+1673677us) spinlock_profiling.cc:243] Waited 3.17 ms on lock 0x7b2c00010f08. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbe0a7e11 00007f3fbe0b7442 00007f3fbf6ffc11 00007f3fbf6c458a 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772
0807 11:10:46.528731 (+ 1126us) log.cc:842] Serialized 26 byte log entry
0807 11:10:46.620382 (+ 91651us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:46.620442 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:46.630601 (+ 10159us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:47.170445 (+539844us) spinlock_profiling.cc:243] Waited 1.37 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:10:47.218027 (+ 47582us) raft_consensus.cc:1671] finished
0807 11:10:47.218055 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:47.221546 (+ 3491us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":427,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84,"spinlock_wait_cycles":10454016,"thread_start_us":71500,"threads_started":1,"wal-append.queue_time_us":265912}
I20260807 11:10:47.252285 26036 rpcz_store.cc:276] 0807 11:10:44.109425 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.221484 (+3112059us) service_pool.cc:227] Handling call
0807 11:10:47.223790 (+ 2306us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.236901 (+ 13111us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.237109 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.254603 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:47.254973 29179 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:47.192512 29191 consensus_queue.cc:237] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:47.259407 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:47.165913 26974 consensus_queue.cc:260] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:47.242172 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:47.266927 26033 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:47.267102 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:47.271252 25917 raft_consensus.cc:2454] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:47.209929 29205 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:47.269806 26986 raft_consensus.cc:2379] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 7: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:47.261868 26985 raft_consensus.cc:2362] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4. Re-sending same reply.
I20260807 11:10:47.275380 29415 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:47.261776 29254 raft_consensus.cc:686] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 5 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:47.275939 26038 raft_consensus.cc:1261] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260807 11:10:47.285640 26037 raft_consensus.cc:2454] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:10:47.267361 26975 raft_consensus.cc:3037] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:47.289456 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:47.292407 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:47.261931 29330 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:47.291846 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:47.297497 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:47.285043 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1649) took 3125 ms. Trace:
I20260807 11:10:47.301620 26038 rpcz_store.cc:276] 0807 11:10:44.152605 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.274987 (+3122382us) service_pool.cc:227] Handling call
0807 11:10:47.275486 (+ 499us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.278109 (+ 2623us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.278284 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.275969 29415 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:47.218887 25654 catalog_manager.cc:5495] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 reported cstate change: config changed from index 2 to 4, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 11:10:47.316337 26035 raft_consensus.cc:3032] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:47.316866 26035 raft_consensus.cc:729] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:47.317370 29330 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:47.318125 29330 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:47.320931 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:47.321777 25921 raft_consensus.cc:3037] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:47.323552 26986 raft_consensus.cc:1203] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Deduplicated request from leader. Original: 3.5->[4.6-4.6] Dedup: 4.6->[]
I20260807 11:10:47.324131 26986 raft_consensus.cc:1226] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260807 11:10:47.325651 26037 raft_consensus.cc:1226] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260807 11:10:47.325889 26938 leader_election.cc:302] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:47.326385 25916 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 240) took 4713 ms. Trace:
I20260807 11:10:47.326642 29500 raft_consensus.cc:482] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:47.327163 29500 raft_consensus.cc:504] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:47.328109 29417 raft_consensus.cc:2781] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:47.328579 29417 raft_consensus.cc:482] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:47.328931 29417 raft_consensus.cc:3037] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:47.329859 29493 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:47.318014 26035 consensus_queue.cc:260] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:47.326339 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1654) took 2539 ms. Trace:
I20260807 11:10:47.328949 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:47.332195 26037 rpcz_store.cc:276] 0807 11:10:44.786714 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.304240 (+2517526us) service_pool.cc:227] Handling call
0807 11:10:47.325262 (+ 21022us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:47.326067 (+ 805us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.326266 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.243448 29373 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:47.326886 25916 rpcz_store.cc:276] 0807 11:10:42.612552 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.182549 (+3569997us) service_pool.cc:227] Handling call
0807 11:10:46.182949 (+ 400us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:46.610350 (+427401us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.326091 (+715741us) spinlock_profiling.cc:243] Waited 384 ms on lock 0x7b540019ffd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:47.326266 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":883100416}
I20260807 11:10:47.337167 29537 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:47.337369 25920 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1669) took 3969 ms (client timeout 2172 ms). Trace:
I20260807 11:10:47.300328 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
W20260807 11:10:47.337890 25920 rpcz_store.cc:269] 0807 11:10:43.367971 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.015931 (+647960us) service_pool.cc:227] Handling call
0807 11:10:47.334591 (+3318660us) spinlock_profiling.cc:243] Waited 872 ms on lock 0x7b54001806d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:47.337163 (+ 2572us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2006567808}
I20260807 11:10:47.338284 26979 raft_consensus.cc:3037] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:47.339365 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:47.340292 25920 raft_consensus.cc:2354] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3.
W20260807 11:10:47.341302 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 87821.3us
I20260807 11:10:47.342128 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 398) took 1178 ms. Trace:
I20260807 11:10:47.342576 26037 rpcz_store.cc:276] 0807 11:10:46.163300 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.341071 (+1177771us) service_pool.cc:227] Handling call
0807 11:10:47.341528 (+ 457us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.341736 (+ 208us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:47.341757 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:47.341813 (+ 56us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:10:47.341862 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.341924 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:47.342030 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.343345 25981 leader_election.cc:302] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:47.276876 29179 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:47.344966 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 399) took 1181 ms. Trace:
I20260807 11:10:47.345367 26037 rpcz_store.cc:276] 0807 11:10:46.163690 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.343910 (+1180220us) service_pool.cc:227] Handling call
0807 11:10:47.344296 (+ 386us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.344630 (+ 334us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:47.344655 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:47.344692 (+ 37us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:10:47.344746 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.344821 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:47.344918 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.346738 29179 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:47.304004 29479 raft_consensus.cc:3037] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:10:47.347481 29445 consensus_queue.cc:260] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:47.348647 29445 raft_consensus.cc:3037] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
W20260807 11:10:47.350772 26936 connection.cc:232] RPC call timeout handler was delayed by -2.8027406549081206s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16190000000000002s for a response
I20260807 11:10:47.350895 26033 raft_consensus.cc:1261] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260807 11:10:47.351341 26033 pending_rounds.cc:76] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9: Aborting all ops after (but not including) 7
W20260807 11:10:47.351401 26936 connection.cc:232] RPC call timeout handler was delayed by -2.4229406549129635s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2041s for a response
I20260807 11:10:47.351673 26033 pending_rounds.cc:97] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9: Aborting uncommitted NO_OP operation due to leader change: 6.8
W20260807 11:10:47.351881 26936 connection.cc:232] RPC call timeout handler was delayed by -2.2839336399920285s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3139s for a response
I20260807 11:10:47.352002 26033 raft_consensus.cc:2866] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260807 11:10:47.352452 26936 connection.cc:232] RPC call timeout handler was delayed by -1.6135146168526262s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1763s for a response
I20260807 11:10:47.353039 29205 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:47.353746 29380 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:47.354027 29418 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:47.271813 29037 consensus_queue.cc:1046] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260807 11:10:47.331065 25918 raft_consensus.cc:3037] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:47.369629 26035 raft_consensus.cc:3037] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
W20260807 11:10:47.369096 26936 connection.cc:232] RPC call timeout handler was delayed by -1.338623110903427s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.284s for a response
I20260807 11:10:47.302807 29013 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:47.333796 29493 raft_consensus.cc:3037] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:47.336625 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1645) took 5710 ms. Trace:
I20260807 11:10:47.372339 26037 raft_consensus.cc:1226] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260807 11:10:47.385836 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 231014us
I20260807 11:10:47.386077 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1657) took 1216 ms. Trace:
I20260807 11:10:47.386515 26037 rpcz_store.cc:276] 0807 11:10:46.169375 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.354457 (+1185082us) service_pool.cc:227] Handling call
0807 11:10:47.372027 (+ 17570us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:47.385847 (+ 13820us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.386030 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.383402 26987 raft_consensus.cc:1226] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 LEADER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 6. Current term is 7. Ops: []
I20260807 11:10:47.388746 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:47.382512 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:47.355141 26041 raft_consensus.cc:1261] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:10:47.390810 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 396) took 3255 ms. Trace:
I20260807 11:10:47.335558 29330 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:47.384940 25925 rpcz_store.cc:276] 0807 11:10:41.625856 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.650224 (+3024368us) service_pool.cc:227] Handling call
0807 11:10:44.907368 (+257144us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:44.908423 (+ 1055us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.336215 (+2427792us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b54001df450. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:47.336414 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":577925376}
I20260807 11:10:47.391736 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:47.395892 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1776) took 2753 ms. Trace:
I20260807 11:10:47.396727 26041 rpcz_store.cc:276] 0807 11:10:44.642077 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.270469 (+2628392us) service_pool.cc:227] Handling call
0807 11:10:47.303697 (+ 33228us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.395642 (+ 91945us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.395831 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.395710 26033 rpcz_store.cc:276] 0807 11:10:44.130364 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.223307 (+3092943us) service_pool.cc:227] Handling call
0807 11:10:47.223744 (+ 437us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.385198 (+161454us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.385509 (+ 311us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40192}
I20260807 11:10:47.383103 26974 raft_consensus.cc:3037] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:47.402827 25931 raft_consensus.cc:3037] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:47.404349 29407 leader_election.cc:288] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:47.405581 25916 raft_consensus.cc:1226] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 6. Current term is 7. Ops: [4.6-6.7]
I20260807 11:10:47.406356 25916 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 243) took 4792 ms. Trace:
I20260807 11:10:47.406414 26936 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:47.406787 25916 rpcz_store.cc:276] 0807 11:10:42.613942 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.378401 (+4764459us) service_pool.cc:227] Handling call
0807 11:10:47.405250 (+ 26849us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:10:47.406006 (+ 756us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.406236 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.409305 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1557) took 4786 ms. Trace:
I20260807 11:10:47.408949 29013 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:47.409543 29418 raft_consensus.cc:504] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:47.402594 29380 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:47.411904 29418 leader_election.cc:288] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:47.418841 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 246) took 4803 ms. Trace:
I20260807 11:10:47.419281 25917 rpcz_store.cc:276] 0807 11:10:42.615130 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.377136 (+4762006us) service_pool.cc:227] Handling call
0807 11:10:47.418233 (+ 41097us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.418493 (+ 260us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:47.418517 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:47.418554 (+ 37us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:10:47.418621 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.418672 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:47.418778 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.420523 29373 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:47.421216 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 778605us
I20260807 11:10:47.421715 29515 leader_election.cc:288] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:47.422726 29254 consensus_queue.cc:237] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:47.425279 26036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1653) took 2751 ms. Trace:
I20260807 11:10:47.425477 26033 raft_consensus.cc:3032] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 11:10:47.425709 26036 rpcz_store.cc:276] 0807 11:10:44.673754 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.270034 (+2596280us) service_pool.cc:227] Handling call
0807 11:10:47.332068 (+ 62034us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.424858 (+ 92790us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:47.424891 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:47.424963 (+ 72us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:47.425032 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.425105 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:47.425220 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.427026 29169 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:47.427461 26036 raft_consensus.cc:1226] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 5. Current term is 6. Ops: []
I20260807 11:10:47.427778 29169 raft_consensus.cc:504] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:47.428930 26038 raft_consensus.cc:1226] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:47.429562 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1655) took 2481 ms. Trace:
I20260807 11:10:47.430007 29169 leader_election.cc:288] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:47.431862 26037 raft_consensus.cc:1261] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 11:10:47.432415 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1785) took 1186 ms. Trace:
I20260807 11:10:47.432736 26037 rpcz_store.cc:276] 0807 11:10:46.246487 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.387650 (+1141163us) service_pool.cc:227] Handling call
0807 11:10:47.431521 (+ 43871us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.432198 (+ 677us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.432367 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.434217 26037 raft_consensus.cc:1261] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 11:10:47.434753 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1789) took 1150 ms. Trace:
I20260807 11:10:47.435039 26037 rpcz_store.cc:276] 0807 11:10:46.284541 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.433560 (+1149019us) service_pool.cc:227] Handling call
0807 11:10:47.433920 (+ 360us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.434507 (+ 587us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.434707 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.436384 26037 raft_consensus.cc:1261] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20260807 11:10:47.438298 29533 log.cc:924] Time spent T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 1.351s user 0.001s sys 0.005s
I20260807 11:10:47.438454 26039 raft_consensus.cc:2454] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:10:47.428135 26036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 401) took 1153 ms. Trace:
I20260807 11:10:47.429857 26038 rpcz_store.cc:276] 0807 11:10:44.948105 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.331455 (+2383350us) service_pool.cc:227] Handling call
0807 11:10:47.428667 (+ 97212us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.429331 (+ 664us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.429488 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.442255 26036 rpcz_store.cc:276] 0807 11:10:46.274806 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.426803 (+1151997us) service_pool.cc:227] Handling call
0807 11:10:47.427215 (+ 412us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.427892 (+ 677us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.428071 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.425966 26033 raft_consensus.cc:729] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:47.450453 25917 raft_consensus.cc:1261] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260807 11:10:47.387323 29500 leader_election.cc:288] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:47.454654 25921 raft_consensus.cc:2454] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
W20260807 11:10:47.450502 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 107347us
I20260807 11:10:47.454094 29037 raft_consensus.cc:2781] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:47.400521 25654 catalog_manager.cc:5495] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 2 to 4, leader changed from eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 4 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:47.461150 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1658) took 4615 ms. Trace:
I20260807 11:10:47.470279 26035 raft_consensus.cc:1261] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260807 11:10:47.470587 25917 rpcz_store.cc:276] 0807 11:10:42.835742 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.420315 (+4584573us) service_pool.cc:227] Handling call
0807 11:10:47.420689 (+ 374us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.450926 (+ 30237us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.451157 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.409708 25932 rpcz_store.cc:276] 0807 11:10:42.599609 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.177594 (+3577985us) service_pool.cc:227] Handling call
0807 11:10:46.608604 (+431010us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:46.609341 (+ 737us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:46.609382 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:46.609443 (+ 61us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:10:46.609484 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:46.609559 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:47.385844 (+776285us) spinlock_profiling.cc:243] Waited 314 ms on lock 0x7b54001a09d0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:47.385944 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":699,"raft.run_cpu_time_us":317,"raft.run_wall_time_us":317,"spinlock_wait_cycles":722070272}
I20260807 11:10:47.411947 29380 leader_election.cc:288] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:47.470677 26035 pending_rounds.cc:76] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9: Aborting all ops after (but not including) 5
I20260807 11:10:47.478080 25918 raft_consensus.cc:2454] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:10:47.481005 26035 pending_rounds.cc:97] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260807 11:10:47.481386 26035 raft_consensus.cc:2866] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:10:47.482851 26038 raft_consensus.cc:1261] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260807 11:10:47.490154 25925 raft_consensus.cc:1261] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260807 11:10:47.494244 29417 raft_consensus.cc:504] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:47.485399 29516 raft_consensus.cc:482] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:47.445288 26978 raft_consensus.cc:2454] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:47.520720 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:47.522046 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:47.523249 25797 raft_consensus.cc:2396] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260807 11:10:47.526504 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:47.527315 25800 raft_consensus.cc:2454] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:47.529191 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:47.529781 26976 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 426) took 4387 ms. Trace:
I20260807 11:10:47.530244 25799 raft_consensus.cc:2379] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 5: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:47.531607 25799 raft_consensus.cc:1226] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:47.530393 26976 rpcz_store.cc:276] 0807 11:10:43.090793 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:43.095425 (+ 4632us) service_pool.cc:227] Handling call
0807 11:10:43.140404 (+ 44979us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:44.206364 (+1065960us) spinlock_profiling.cc:243] Waited 4.47 ms on lock 0x7b180027a240. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbc79c764 00007f3fbc79fb11 00007f3fbc79fac9 00007f3fbc79fa59 00007f3fbc79fa21 00007f3fbc79ed1d 00007f3fc5a00204 00007f3fc5a00039 00007f3fbc77edc6
0807 11:10:44.784369 (+578005us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x7b180027a240. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbc77e345 00007f3fc55b5a39 00007f3fbc79b8bb 00007f3fbc799cbf 00007f3fbc799155 00007f3fbf6bc3d0 00007f3fbf6bcf6b 00007f3fbf6c778b 00007f3fbf6c3edf
0807 11:10:44.932274 (+147905us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:44.939864 (+ 7590us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:44.997058 (+ 57194us) log.cc:842] Serialized 26 byte log entry
0807 11:10:45.465390 (+468332us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:45.465468 (+ 78us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:45.465597 (+ 129us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:47.377412 (+1911815us) spinlock_profiling.cc:243] Waited 18.5 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:10:47.440237 (+ 62825us) raft_consensus.cc:1671] finished
0807 11:10:47.445467 (+ 5230us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:47.475141 (+ 29674us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":57310,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":8137,"spinlock_wait_cycles":90212352,"thread_start_us":537513,"threads_started":2,"wal-append.queue_time_us":889708}
I20260807 11:10:47.507774 25932 raft_consensus.cc:1261] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 11:10:47.511051 25917 raft_consensus.cc:1261] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 11:10:47.511497 26938 leader_election.cc:302] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:47.511955 29029 raft_consensus.cc:2726] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 11:10:47.512077 29267 consensus_queue.cc:1035] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260807 11:10:47.478432 29330 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:47.517555 25921 raft_consensus.cc:1226] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6. Ops: []
I20260807 11:10:47.517976 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:47.554973 25796 raft_consensus.cc:2454] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 7.
I20260807 11:10:47.500983 25654 catalog_manager.cc:5495] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 1 to 3, leader changed from 4fd6381544574ddca559ce63b9243efc (127.23.120.130) to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 3 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260807 11:10:47.533375 25801 raft_consensus.cc:1226] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:47.533984 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:47.539420 29417 leader_election.cc:288] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:47.502682 29500 raft_consensus.cc:482] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:47.583307 25808 raft_consensus.cc:2454] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
W20260807 11:10:47.545641 29527 log.cc:924] Time spent T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 1.517s user 0.000s sys 0.006s
I20260807 11:10:47.528328 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:47.583091 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:47.586404 25800 raft_consensus.cc:2396] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 11:10:47.524377 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:47.505556 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1656) took 4683 ms. Trace:
I20260807 11:10:47.587327 25925 rpcz_store.cc:276] 0807 11:10:42.822217 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.441219 (+4619002us) service_pool.cc:227] Handling call
0807 11:10:47.479224 (+ 38005us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.490618 (+ 11394us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.505486 (+ 14868us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.553040 26041 raft_consensus.cc:3032] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:47.589421 26033 consensus_queue.cc:260] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:47.557400 26984 raft_consensus.cc:2354] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6.
I20260807 11:10:47.513576 25798 raft_consensus.cc:1261] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 11:10:47.573762 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1577) took 3016 ms. Trace:
I20260807 11:10:47.591656 25932 rpcz_store.cc:276] 0807 11:10:44.542450 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.480090 (+2937640us) service_pool.cc:227] Handling call
0807 11:10:47.507390 (+ 27300us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.558745 (+ 51355us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.558977 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.571744 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1567) took 3460 ms. Trace:
I20260807 11:10:47.592876 25917 rpcz_store.cc:276] 0807 11:10:44.099330 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.477177 (+3377847us) service_pool.cc:227] Handling call
0807 11:10:47.510635 (+ 33458us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.558940 (+ 48305us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.559827 (+ 887us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.593995 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:47.575170 26035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1775) took 3047 ms. Trace:
I20260807 11:10:47.518594 29516 raft_consensus.cc:3037] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:47.582535 25806 raft_consensus.cc:1226] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260807 11:10:47.601838 25654 catalog_manager.cc:5495] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 3 to 4, leader changed from eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 4 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 11:10:47.583881 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1565) took 3825 ms. Trace:
I20260807 11:10:47.555994 29013 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:47.595124 26987 raft_consensus.cc:2396] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260807 11:10:47.595118 26035 rpcz_store.cc:276] 0807 11:10:44.515665 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.263941 (+2748276us) service_pool.cc:227] Handling call
0807 11:10:47.316114 (+ 52173us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.562657 (+246543us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.562862 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.589809 26041 raft_consensus.cc:729] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
W20260807 11:10:47.598214 25869 connection.cc:232] RPC call timeout handler was delayed by -1.8632178090047091s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1758s for a response
I20260807 11:10:47.604521 25921 rpcz_store.cc:276] 0807 11:10:43.758195 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.473935 (+3715740us) service_pool.cc:227] Handling call
0807 11:10:47.517225 (+ 43290us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.583567 (+ 66342us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.583817 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.606060 25750 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:47.585816 29169 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:47.613505 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 50136.2us
I20260807 11:10:47.616132 25654 catalog_manager.cc:5495] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 1 to 3, leader changed from b22d69db45194908ad3d345eba9badd9 (127.23.120.131) to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 3 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } }
I20260807 11:10:47.611469 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:47.599467 29417 raft_consensus.cc:2781] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:47.622658 29569 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:47.563838 29261 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:47.623497 25799 raft_consensus.cc:1226] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260807 11:10:47.623570 29569 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:47.626194 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:47.626748 25919 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:47.609799 29217 consensus_queue.cc:1046] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260807 11:10:47.627382 29169 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:47.621593 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:47.628583 29217 raft_consensus.cc:3032] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:47.629029 25800 raft_consensus.cc:2396] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 11:10:47.618995 25925 raft_consensus.cc:2454] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:47.622901 29417 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:47.620430 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:47.629042 29217 raft_consensus.cc:729] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:47.634202 26033 raft_consensus.cc:3037] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:47.630646 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:47.639990 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:47.640165 25925 raft_consensus.cc:1226] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6. Ops: []
I20260807 11:10:47.640794 29558 raft_consensus.cc:2726] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 11:10:47.640327 29261 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:47.641211 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1578) took 3055 ms. Trace:
I20260807 11:10:47.641773 25925 rpcz_store.cc:276] 0807 11:10:44.585715 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.634685 (+3048970us) service_pool.cc:227] Handling call
0807 11:10:47.639927 (+ 5242us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.640871 (+ 944us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.641119 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.640056 29217 consensus_queue.cc:260] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:47.643944 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:47.645124 29217 raft_consensus.cc:3037] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:47.591240 29500 raft_consensus.cc:504] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:47.645053 25921 raft_consensus.cc:2454] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:47.650578 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:47.655054 29261 consensus_queue.cc:1046] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 11:10:47.655911 25932 raft_consensus.cc:1261] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 11:10:47.656592 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1686) took 2490 ms. Trace:
I20260807 11:10:47.656936 25932 rpcz_store.cc:276] 0807 11:10:45.165990 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.655208 (+2489218us) service_pool.cc:227] Handling call
0807 11:10:47.655586 (+ 378us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.656342 (+ 756us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.656513 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.658355 25932 raft_consensus.cc:1261] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260807 11:10:47.656055 29261 raft_consensus.cc:3032] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Stepping down as leader of term 6
I20260807 11:10:47.658639 25932 pending_rounds.cc:76] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc: Aborting all ops after (but not including) 2
I20260807 11:10:47.658807 29261 raft_consensus.cc:729] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:47.626034 29569 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:47.659749 29261 consensus_queue.cc:260] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:47.661105 29261 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:47.658937 25932 pending_rounds.cc:97] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260807 11:10:47.662034 25932 raft_consensus.cc:2866] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:10:47.664860 29558 raft_consensus.cc:482] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:47.669061 29558 raft_consensus.cc:504] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:47.671371 29558 leader_election.cc:288] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:47.672021 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1771) took 3562 ms. Trace:
I20260807 11:10:47.672400 26034 rpcz_store.cc:276] 0807 11:10:44.109106 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.172204 (+3063098us) service_pool.cc:227] Handling call
0807 11:10:47.172756 (+ 552us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:47.173479 (+ 723us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:47.173514 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:47.174155 (+ 641us) log.cc:842] Serialized 26 byte log entry
0807 11:10:47.476391 (+302236us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:47.476435 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.476517 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:47.671808 (+195291us) raft_consensus.cc:1671] finished
0807 11:10:47.671829 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:47.671961 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":150,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":241,"thread_start_us":298505,"threads_started":1,"wal-append.queue_time_us":429783}
I20260807 11:10:47.664281 25925 raft_consensus.cc:1261] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 11:10:47.674513 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1682) took 2751 ms. Trace:
I20260807 11:10:47.674880 25925 rpcz_store.cc:276] 0807 11:10:44.923160 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.652526 (+2729366us) service_pool.cc:227] Handling call
0807 11:10:47.663841 (+ 11315us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.674277 (+ 10436us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.674463 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.636271 29417 raft_consensus.cc:3037] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:47.690802 25925 raft_consensus.cc:1261] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:10:47.692433 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1688) took 1135 ms. Trace:
I20260807 11:10:47.692756 25925 rpcz_store.cc:276] 0807 11:10:46.556928 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.687180 (+1130252us) service_pool.cc:227] Handling call
0807 11:10:47.687554 (+ 374us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.692208 (+ 4654us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.692386 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.651575 26041 consensus_queue.cc:260] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:47.692335 25654 catalog_manager.cc:5495] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 1 to 3, leader changed from 606d49e053a346e486f48cb31832b815 (127.23.120.129) to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 3 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260807 11:10:47.613171 29380 consensus_queue.cc:1046] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260807 11:10:47.702169 25654 catalog_manager.cc:5495] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } }
W20260807 11:10:47.704319 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 128219us
I20260807 11:10:47.617111 29330 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:47.706427 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1788) took 1446 ms. Trace:
I20260807 11:10:47.708232 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:47.709813 26033 rpcz_store.cc:276] 0807 11:10:46.259676 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.424814 (+1165138us) service_pool.cc:227] Handling call
0807 11:10:47.425267 (+ 453us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.705958 (+280691us) raft_consensus.cc:1520] Early marking committed up to index 2
0807 11:10:47.706011 (+ 53us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:47.706074 (+ 63us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:47.706139 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.706226 (+ 87us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:47.706364 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":184,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":8849}
I20260807 11:10:47.710788 25750 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:47.711630 29558 raft_consensus.cc:482] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:47.720867 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:47.718443 25925 raft_consensus.cc:1261] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260807 11:10:47.722191 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1689) took 1164 ms. Trace:
I20260807 11:10:47.722589 25925 rpcz_store.cc:276] 0807 11:10:46.557690 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.693441 (+1135751us) service_pool.cc:227] Handling call
0807 11:10:47.714287 (+ 20846us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.721903 (+ 7616us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.722131 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":663808}
I20260807 11:10:47.727317 25917 raft_consensus.cc:1226] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: [3.2-4.3]
I20260807 11:10:47.728222 25869 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:47.730207 29342 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:47.730365 29029 consensus_queue.cc:1046] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:10:47.731123 29173 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:47.731557 25919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1695) took 1092 ms. Trace:
I20260807 11:10:47.731709 29278 raft_consensus.cc:3032] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:47.731974 25919 rpcz_store.cc:276] 0807 11:10:46.638945 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.242916 (+603971us) service_pool.cc:227] Handling call
0807 11:10:47.731396 (+488480us) spinlock_profiling.cc:243] Waited 380 ms on lock 0x7b54000366d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:47.731504 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":873007488}
I20260807 11:10:47.732357 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1687) took 2509 ms. Trace:
I20260807 11:10:47.730819 29342 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:47.734561 25918 raft_consensus.cc:1226] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:47.734153 25932 rpcz_store.cc:276] 0807 11:10:45.223120 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.657653 (+2434533us) service_pool.cc:227] Handling call
0807 11:10:47.658031 (+ 378us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.732116 (+ 74085us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.732299 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.741917 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1684) took 2705 ms. Trace:
I20260807 11:10:47.742470 25921 rpcz_store.cc:276] 0807 11:10:45.036922 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.682490 (+2645568us) service_pool.cc:227] Handling call
0807 11:10:47.726942 (+ 44452us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.741451 (+ 14509us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:47.741493 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:47.741554 (+ 61us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:47.741611 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.741686 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:47.741847 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.735906 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1690) took 1177 ms. Trace:
I20260807 11:10:47.745905 25917 rpcz_store.cc:276] 0807 11:10:46.558202 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.700922 (+1142720us) service_pool.cc:227] Handling call
0807 11:10:47.718722 (+ 17800us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:10:47.735578 (+ 16856us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.735841 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.688572 29510 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:47.762991 25919 raft_consensus.cc:1261] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 11:10:47.765502 25919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1699) took 1118 ms. Trace:
I20260807 11:10:47.765861 25919 rpcz_store.cc:276] 0807 11:10:46.647172 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.733547 (+1086375us) service_pool.cc:227] Handling call
0807 11:10:47.759528 (+ 25981us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.765240 (+ 5712us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.765441 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:47.767609 25932 raft_consensus.cc:1261] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 6. (term mismatch)
I20260807 11:10:47.768124 25932 pending_rounds.cc:76] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc: Aborting all ops after (but not including) 5
I20260807 11:10:47.768399 25932 pending_rounds.cc:97] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260807 11:10:47.768707 25932 raft_consensus.cc:2866] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:10:47.770576 25917 raft_consensus.cc:1261] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 11:10:47.732184 29278 raft_consensus.cc:729] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
W20260807 11:10:47.772815 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 53641.9us
I20260807 11:10:47.772336 25654 catalog_manager.cc:5495] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 1 to 4, leader changed from b22d69db45194908ad3d345eba9badd9 (127.23.120.131) to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 4 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } }
I20260807 11:10:47.741215 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:47.781719 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:47.780589 25931 raft_consensus.cc:2454] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:10:47.783051 25925 raft_consensus.cc:1261] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 11:10:47.787282 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1694) took 1167 ms. Trace:
I20260807 11:10:47.787731 25925 rpcz_store.cc:276] 0807 11:10:46.619838 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.723577 (+1103739us) service_pool.cc:227] Handling call
0807 11:10:47.759558 (+ 35981us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.786965 (+ 27407us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.787214 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14080}
I20260807 11:10:47.731752 29173 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:47.792062 28964 raft_consensus.cc:2726] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:47.792891 28761 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:47.793682 28761 raft_consensus.cc:504] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:47.794301 29415 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:47.795953 28761 leader_election.cc:288] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:47.796367 29415 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:47.796686 25803 raft_consensus.cc:2396] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260807 11:10:47.796898 29569 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:47.797516 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:47.797431 29569 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:47.799154 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:47.792831 25872 leader_election.cc:302] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:47.799893 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:47.783529 26984 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50960 (request call id 356) took 1014 ms. Trace:
I20260807 11:10:47.800695 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:47.763399 29558 raft_consensus.cc:504] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:47.800280 29569 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:47.782137 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1648) took 3641 ms. Trace:
I20260807 11:10:47.801412 26979 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 2.
W20260807 11:10:47.914649 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 424362us
I20260807 11:10:47.911902 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:48.007110 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:47.937673 25981 leader_election.cc:302] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:47.819175 25750 leader_election.cc:302] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:47.953229 26041 raft_consensus.cc:3037] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:47.881090 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:47.801499 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:47.877537 29516 raft_consensus.cc:504] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:48.008843 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 310017us
I20260807 11:10:47.937968 25918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1677) took 3372 ms. Trace:
I20260807 11:10:48.048978 25918 rpcz_store.cc:276] 0807 11:10:44.566004 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.490550 (+2924546us) service_pool.cc:227] Handling call
0807 11:10:47.519693 (+ 29143us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.738472 (+218779us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.937692 (+199220us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b5400408654. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:47.937905 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":493669248}
I20260807 11:10:47.826726 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:48.006512 26984 rpcz_store.cc:276] 0807 11:10:46.769154 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.368242 (+599088us) service_pool.cc:227] Handling call
0807 11:10:47.781062 (+412820us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b54004d0b54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:47.783455 (+ 2393us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":348346240}
I20260807 11:10:47.875270 25925 raft_consensus.cc:1261] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260807 11:10:47.825326 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:48.005415 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:48.068334 26035 raft_consensus.cc:2396] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 6 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 11:10:47.810813 25931 raft_consensus.cc:1261] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260807 11:10:47.880087 25919 raft_consensus.cc:1226] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:48.044384 26038 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 7.
I20260807 11:10:47.849982 25654 catalog_manager.cc:5495] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 1 to 3, leader changed from 606d49e053a346e486f48cb31832b815 (127.23.120.129) to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 3 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
W20260807 11:10:48.072851 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 283582us
I20260807 11:10:47.909953 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:47.827562 26975 consensus_meta.cc:287] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.527s user 0.011s sys 0.000s
I20260807 11:10:47.825426 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:48.048362 26033 raft_consensus.cc:2454] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:48.007071 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:48.053877 26979 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50960 (request call id 353) took 1280 ms. Trace:
I20260807 11:10:48.054092 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:47.782606 29307 consensus_queue.cc:1046] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.274s
I20260807 11:10:47.790515 29037 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:48.080068 25800 raft_consensus.cc:2454] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:10:48.058284 29516 leader_election.cc:288] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:48.059044 29217 consensus_queue.cc:1035] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 11:10:48.060262 26037 raft_consensus.cc:2454] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:10:48.061383 26978 raft_consensus.cc:1226] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 1. Current term is 2. Ops: []
I20260807 11:10:47.942960 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1770) took 3834 ms. Trace:
I20260807 11:10:47.959403 29267 raft_consensus.cc:3032] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 11:10:48.076720 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:47.790607 26974 raft_consensus.cc:1261] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 11:10:48.056732 29394 consensus_queue.cc:1035] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 11:10:48.101684 26034 raft_consensus.cc:2396] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260807 11:10:48.081859 25750 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:48.077446 29589 raft_consensus.cc:2781] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:48.103140 29589 raft_consensus.cc:482] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:48.083989 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:47.977710 29500 leader_election.cc:288] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:47.958724 29173 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:48.012053 26046 rpcz_store.cc:276] 0807 11:10:44.150425 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.230454 (+3080029us) service_pool.cc:227] Handling call
0807 11:10:47.230843 (+ 389us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.231089 (+ 246us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 11:10:47.555524 (+324435us) log.cc:842] Serialized 14 byte log entry
0807 11:10:47.781788 (+226264us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:47.781840 (+ 52us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:47.781905 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.781974 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:47.782081 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":459444,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":2522,"raft.queue_time_us":82,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":19108,"thread_start_us":535198,"threads_started":2,"wal-append.queue_time_us":443514}
I20260807 11:10:47.958464 29572 raft_consensus.cc:2726] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:48.097829 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:48.095703 26042 rpcz_store.cc:276] 0807 11:10:44.108801 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.134829 (+3026028us) service_pool.cc:227] Handling call
0807 11:10:47.215358 (+ 80529us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:47.386159 (+170801us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:47.386189 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:47.412374 (+ 26185us) log.cc:842] Serialized 26 byte log entry
0807 11:10:47.734390 (+322016us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:47.734640 (+ 250us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.938375 (+203735us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b5400312c54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:47.938438 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:47.938499 (+ 61us) raft_consensus.cc:1671] finished
0807 11:10:47.938517 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:47.942883 (+ 4366us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4716,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":54474,"spinlock_wait_cycles":320286080,"thread_start_us":320676,"threads_started":1,"wal-append.queue_time_us":385319}
I20260807 11:10:48.084362 29037 raft_consensus.cc:3037] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:48.111682 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:48.098577 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:48.100412 26936 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.021s sys 0.050s
I20260807 11:10:47.958259 29169 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:48.097796 25803 raft_consensus.cc:2454] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:48.103679 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:48.113332 26038 raft_consensus.cc:2379] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 2: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:48.113505 26034 raft_consensus.cc:2379] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 4: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:48.113710 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:39730 (request call id 691) took 4157 ms. Trace:
I20260807 11:10:48.104091 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:48.105729 26039 raft_consensus.cc:2379] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 6: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
W20260807 11:10:48.091192 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 454192us
I20260807 11:10:48.022811 29330 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:48.080899 26980 raft_consensus.cc:2454] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:48.085516 25654 catalog_manager.cc:5495] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } }
I20260807 11:10:48.103541 29589 raft_consensus.cc:3037] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:48.114073 25917 rpcz_store.cc:276] 0807 11:10:43.939974 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.851571 (+3911597us) service_pool.cc:227] Handling call
0807 11:10:48.097616 (+246045us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:48.097807 29267 raft_consensus.cc:729] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
W20260807 11:10:48.112779 26936 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.524s user 0.021s sys 0.050s
I20260807 11:10:48.085428 29342 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:48.120433 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:48.120693 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 566224us
I20260807 11:10:48.104909 29205 raft_consensus.cc:2726] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260807 11:10:48.124130 29587 log.cc:924] Time spent T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.204s user 0.001s sys 0.001s
I20260807 11:10:48.124512 29267 consensus_queue.cc:260] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:48.125571 29323 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260807 11:10:48.125777 29267 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:48.128064 29323 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:48.129556 25926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1641) took 6504 ms. Trace:
I20260807 11:10:48.130052 25926 rpcz_store.cc:276] 0807 11:10:41.624664 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:44.635739 (+3011075us) service_pool.cc:227] Handling call
0807 11:10:44.647756 (+ 12017us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:44.648102 (+ 346us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:44.648135 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:47.341305 (+2693170us) spinlock_profiling.cc:243] Waited 28.7 ms on lock 0x7b2c0004b3f8. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbe0a7e11 00007f3fbe0b7442 00007f3fbf6ffc11 00007f3fbf6c458a 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772
0807 11:10:47.341872 (+ 567us) log.cc:842] Serialized 26 byte log entry
0807 11:10:47.700255 (+358383us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:47.700302 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.707926 (+ 7624us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:48.128972 (+421046us) raft_consensus.cc:1671] finished
0807 11:10:48.129001 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:48.129335 (+ 334us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4111,"raft.queue_time_us":11745,"raft.run_cpu_time_us":1048,"raft.run_wall_time_us":28089,"spinlock_wait_cycles":66104576,"thread_start_us":357957,"threads_started":1,"wal-append.queue_time_us":459006}
W20260807 11:10:48.134081 29588 log.cc:924] Time spent T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.129s user 0.000s sys 0.000s
I20260807 11:10:48.110739 29445 raft_consensus.cc:2781] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:48.138804 29445 raft_consensus.cc:482] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:48.139166 29445 raft_consensus.cc:3037] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:48.142542 26042 raft_consensus.cc:2379] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 3: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:48.115253 29515 raft_consensus.cc:3032] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:48.145146 29515 raft_consensus.cc:729] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:48.145785 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:48.146427 29515 consensus_queue.cc:260] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:48.148316 29515 raft_consensus.cc:3037] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:48.114809 29516 raft_consensus.cc:2726] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:10:48.149581 26034 raft_consensus.cc:1226] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 5. Current term is 6. Ops: []
I20260807 11:10:48.150938 26034 raft_consensus.cc:3032] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 11:10:48.151278 26034 raft_consensus.cc:729] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:48.152046 26034 consensus_queue.cc:260] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:48.165208 26034 raft_consensus.cc:3037] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:48.165354 25869 leader_election.cc:302] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:48.167215 25869 leader_election.cc:302] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:48.107024 29510 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:48.169739 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:48.170538 26038 raft_consensus.cc:3037] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 6
I20260807 11:10:48.170814 25916 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1657) took 5348 ms. Trace:
I20260807 11:10:48.171224 25916 rpcz_store.cc:276] 0807 11:10:42.822532 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.407677 (+4585145us) service_pool.cc:227] Handling call
0807 11:10:47.408134 (+ 457us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:47.449231 (+ 41097us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:47.449271 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:47.547809 (+ 98538us) log.cc:842] Serialized 26 byte log entry
0807 11:10:47.878028 (+330219us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:10:47.878088 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:47.878181 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:48.170555 (+292374us) raft_consensus.cc:1671] finished
0807 11:10:48.170575 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:48.170710 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":94672,"raft.queue_time_us":37880,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":28204,"thread_start_us":278243,"threads_started":1,"wal-append.queue_time_us":535309}
I20260807 11:10:48.080374 26979 rpcz_store.cc:276] 0807 11:10:46.767875 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.265148 (+497273us) service_pool.cc:227] Handling call
0807 11:10:48.021798 (+756650us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:48.161162 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:48.173875 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 262336us
I20260807 11:10:48.177495 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:48.177646 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:48.178128 26036 raft_consensus.cc:2454] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:48.179335 26033 raft_consensus.cc:2379] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 5: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:48.119117 26975 raft_consensus.cc:1261] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 11:10:48.173309 29394 consensus_queue.cc:1035] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260807 11:10:48.183535 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 101746us
W20260807 11:10:48.121277 25752 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 2.490s (SENT) [suppressed 14 similar messages]
I20260807 11:10:48.118680 26974 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 441) took 1494 ms. Trace:
I20260807 11:10:47.949961 29278 consensus_queue.cc:260] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:48.185266 25872 leader_election.cc:302] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:48.183564 25654 catalog_manager.cc:5495] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260807 11:10:48.185832 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:48.182322 29013 consensus_queue.cc:1035] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20260807 11:10:48.186673 26974 rpcz_store.cc:276] 0807 11:10:46.621726 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.628592 (+ 6866us) service_pool.cc:227] Handling call
0807 11:10:46.629112 (+ 520us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:48.101276 (+1472164us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:48.115741 (+ 14465us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:48.189059 26035 raft_consensus.cc:2454] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:48.190737 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:48.191803 26034 raft_consensus.cc:1261] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 11:10:48.180214 29169 raft_consensus.cc:2726] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:48.128587 29323 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:48.188531 29267 consensus_queue.cc:1159] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4fd6381544574ddca559ce63b9243efc" responder_term: 5 status { last_received { term: 4 index: 6 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 4. Current term is 5. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260807 11:10:48.195202 26936 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:48.196259 25750 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
W20260807 11:10:48.196812 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 76320.7us
I20260807 11:10:48.198393 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:48.188244 29493 consensus_meta.cc:287] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.700s user 0.011s sys 0.000s
I20260807 11:10:48.197471 25920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1664) took 4987 ms. Trace:
I20260807 11:10:48.204864 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:48.181288 29330 raft_consensus.cc:2726] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 11:10:48.200757 29500 raft_consensus.cc:2781] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:48.200096 25654 catalog_manager.cc:5495] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 3 to 4, leader changed from eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 4 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:10:48.203506 25869 leader_election.cc:302] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:48.206739 26035 raft_consensus.cc:1261] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260807 11:10:48.220017 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:48.198725 26975 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 444) took 1575 ms. Trace:
I20260807 11:10:48.221053 26034 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:48.214634 25750 leader_election.cc:302] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:48.226482 26035 raft_consensus.cc:3037] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:48.211135 29510 raft_consensus.cc:2781] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:48.223261 29261 consensus_queue.cc:1035] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:10:48.210814 29593 raft_consensus.cc:2726] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 11:10:48.215672 29500 raft_consensus.cc:482] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:48.227969 25654 catalog_manager.cc:5495] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260807 11:10:48.216557 29278 raft_consensus.cc:3037] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:48.230908 25869 leader_election.cc:302] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:48.217147 29169 raft_consensus.cc:2781] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:48.205173 25920 rpcz_store.cc:276] 0807 11:10:43.210089 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.437063 (+4226974us) service_pool.cc:227] Handling call
0807 11:10:47.437438 (+ 375us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.437763 (+ 325us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 11:10:47.732964 (+295201us) log.cc:842] Serialized 14 byte log entry
0807 11:10:48.184404 (+451440us) log.cc:842] Serialized 14 byte log entry
0807 11:10:48.197155 (+ 12751us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:48.197214 (+ 59us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:48.197262 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:48.197322 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:48.197427 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7691,"prepare.queue_time_us":387265,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":85,"raft.queue_time_us":11262,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":1888,"spinlock_wait_cycles":3516160,"thread_start_us":681141,"threads_started":2}
W20260807 11:10:48.243331 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 53122.8us
I20260807 11:10:48.209887 29493 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:48.240777 29169 raft_consensus.cc:482] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:48.246287 29323 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:48.246526 29169 raft_consensus.cc:3037] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:48.247138 29037 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:48.233508 29510 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:48.253340 25981 leader_election.cc:302] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:48.237291 29342 raft_consensus.cc:2726] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 11:10:48.238241 29500 raft_consensus.cc:3037] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:48.259914 26037 raft_consensus.cc:1226] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:48.256500 29254 raft_consensus.cc:2726] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:48.262758 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:48.264339 25654 catalog_manager.cc:5495] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 1 to 3, leader changed from 4fd6381544574ddca559ce63b9243efc (127.23.120.130) to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 3 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } }
I20260807 11:10:48.222250 26975 rpcz_store.cc:276] 0807 11:10:46.623219 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:46.625038 (+ 1819us) service_pool.cc:227] Handling call
0807 11:10:46.625869 (+ 831us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:48.183594 (+1557725us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:48.195983 (+ 12389us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:48.255353 29510 raft_consensus.cc:3037] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:48.254031 29558 leader_election.cc:288] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:48.267751 25750 leader_election.cc:302] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:48.267143 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1703) took 1003 ms. Trace:
I20260807 11:10:48.278568 29037 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:48.265592 29205 raft_consensus.cc:2726] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:48.279763 29254 consensus_queue.cc:1035] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 11:10:48.281378 25869 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:48.284462 26033 raft_consensus.cc:2454] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:10:48.286063 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:48.273824 25654 catalog_manager.cc:5495] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } }
I20260807 11:10:48.285907 26938 leader_election.cc:302] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:48.291785 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1787) took 2044 ms. Trace:
I20260807 11:10:48.293527 25654 catalog_manager.cc:5495] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 1 to 3, leader changed from b22d69db45194908ad3d345eba9badd9 (127.23.120.131) to 606d49e053a346e486f48cb31832b815 (127.23.120.129). New cstate: current_term: 3 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } }
I20260807 11:10:48.256690 29445 raft_consensus.cc:504] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:48.292176 26041 rpcz_store.cc:276] 0807 11:10:46.247150 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.403627 (+1156477us) service_pool.cc:227] Handling call
0807 11:10:47.491378 (+ 87751us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:47.952956 (+461578us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b54003bfc20. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf65240d 00007f3fbf640050 00007f3fbf6a3aba 00007f3fbf6bb3af 00007f3fbf6bcb4e 00007f3fbf6c6c89 00007f3fbf6c7556 00007f3fbf6c3edf 00007f3fbf6c399e 00007f3fc46052e9
0807 11:10:48.270126 (+317170us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:48.270157 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:48.270395 (+ 238us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:48.291547 (+ 21152us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:48.291613 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:48.291725 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":697,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":7075,"spinlock_wait_cycles":423632768}
I20260807 11:10:48.311928 26038 raft_consensus.cc:2454] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:10:48.314453 26938 leader_election.cc:302] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:48.315510 26035 raft_consensus.cc:1261] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 11:10:48.298933 28964 raft_consensus.cc:2781] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:10:48.313256 25654 catalog_manager.cc:5495] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 1 to 3, leader changed from 606d49e053a346e486f48cb31832b815 (127.23.120.129) to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 3 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260807 11:10:48.299468 29037 raft_consensus.cc:2781] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:10:48.316792 26034 raft_consensus.cc:2454] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:10:48.322547 25869 leader_election.cc:302] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:48.323891 28964 raft_consensus.cc:686] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:48.324980 28964 consensus_queue.cc:237] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:48.311687 29342 consensus_queue.cc:1035] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.017s
W20260807 11:10:48.317272 29417 consensus_meta.cc:287] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.605s user 0.008s sys 0.003s
I20260807 11:10:48.330183 29417 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:48.332355 29417 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:48.333406 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:48.334105 26034 raft_consensus.cc:3037] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:48.338011 29572 raft_consensus.cc:2726] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 11:10:48.338847 29572 raft_consensus.cc:2741] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260807 11:10:48.334678 29394 raft_consensus.cc:2781] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election won for term 6
W20260807 11:10:48.341324 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 220053us
I20260807 11:10:48.341343 25750 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:48.342628 25654 catalog_manager.cc:5495] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 1 to 3, leader changed from 4fd6381544574ddca559ce63b9243efc (127.23.120.130) to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 3 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260807 11:10:48.317589 29558 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:48.322053 29037 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:48.334164 29445 leader_election.cc:288] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:48.358898 25750 leader_election.cc:302] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:48.365362 29415 raft_consensus.cc:2781] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:10:48.366058 26034 raft_consensus.cc:2454] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:10:48.367542 25654 catalog_manager.cc:5495] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 1 to 3, leader changed from 606d49e053a346e486f48cb31832b815 (127.23.120.129) to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 3 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } }
I20260807 11:10:48.371347 26936 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:48.345613 29394 raft_consensus.cc:686] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:48.360258 29037 raft_consensus.cc:3037] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:48.410779 25654 catalog_manager.cc:5495] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 2 to 3, leader changed from 4fd6381544574ddca559ce63b9243efc (127.23.120.130) to b22d69db45194908ad3d345eba9badd9 (127.23.120.131). New cstate: current_term: 3 leader_uuid: "b22d69db45194908ad3d345eba9badd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260807 11:10:48.374948 26938 leader_election.cc:302] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:10:48.413218 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 154043us
I20260807 11:10:48.418639 25654 catalog_manager.cc:5495] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "606d49e053a346e486f48cb31832b815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } }
I20260807 11:10:48.422745 25872 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:48.423871 29445 raft_consensus.cc:2726] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:48.424727 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:48.425035 25872 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:48.425355 25916 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 7.
I20260807 11:10:48.425535 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:48.426203 25872 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:48.426573 25806 raft_consensus.cc:2379] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 6: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:48.426693 29445 raft_consensus.cc:2781] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 11:10:48.427131 29330 raft_consensus.cc:2781] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:10:48.427263 29445 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:48.427537 29330 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:48.428011 29330 raft_consensus.cc:3037] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:48.429384 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:48.430476 25916 raft_consensus.cc:2396] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260807 11:10:48.421290 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:48.433230 25916 raft_consensus.cc:1226] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 6. Current term is 7. Ops: [4.6-6.7]
I20260807 11:10:48.427660 29445 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:10:48.433718 25796 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 7.
I20260807 11:10:48.441557 25916 pending_rounds.cc:76] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc: Aborting all ops after (but not including) 3
I20260807 11:10:48.441184 29445 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:48.442651 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:48.419409 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 44567.5us
I20260807 11:10:48.443490 26037 raft_consensus.cc:2396] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 11:10:48.443846 29445 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:48.440307 29330 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:48.441900 25916 pending_rounds.cc:97] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20260807 11:10:48.445159 25916 raft_consensus.cc:2866] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:10:48.448216 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:48.449272 26037 raft_consensus.cc:2379] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 4: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:48.450142 25931 raft_consensus.cc:1226] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260807 11:10:48.453135 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:48.453897 25931 raft_consensus.cc:3037] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:48.455891 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:48.446455 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:48.458959 25926 raft_consensus.cc:2454] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:10:48.457535 26037 raft_consensus.cc:1261] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 11:10:48.373191 29415 raft_consensus.cc:686] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
W20260807 11:10:48.465494 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 141944us
I20260807 11:10:48.467103 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:48.467201 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:48.464161 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:48.371225 29516 raft_consensus.cc:2781] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:48.469895 26034 raft_consensus.cc:2379] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 5: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:48.469799 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:48.470440 26041 raft_consensus.cc:3037] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:48.360478 29558 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:48.471611 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:48.446431 29330 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:48.472491 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:48.473999 26038 raft_consensus.cc:1261] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20260807 11:10:48.414920 29493 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:10:48.475405 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 60224us
I20260807 11:10:48.476969 26038 pending_rounds.cc:76] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9: Aborting all ops after (but not including) 1
I20260807 11:10:48.477245 26038 pending_rounds.cc:97] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260807 11:10:48.477525 26038 raft_consensus.cc:2866] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:10:48.478314 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:48.476055 25931 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:10:48.479351 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:48.404874 29593 consensus_queue.cc:1046] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.065s
I20260807 11:10:48.479947 26033 raft_consensus.cc:3037] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:48.412107 29013 consensus_queue.cc:1035] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
I20260807 11:10:48.405501 29417 consensus_queue.cc:1035] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.094s
I20260807 11:10:48.469566 29380 raft_consensus.cc:2781] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:10:48.482053 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:48.466868 29267 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:48.469060 29597 raft_consensus.cc:2781] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 11:10:48.471517 29516 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:48.417683 29342 consensus_queue.cc:1035] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
I20260807 11:10:48.478041 25920 raft_consensus.cc:2454] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
W20260807 11:10:48.485716 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 163267us
I20260807 11:10:48.485292 29415 consensus_queue.cc:237] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:48.484203 25752 leader_election.cc:302] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:48.498867 25869 leader_election.cc:302] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:48.486269 29267 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:48.515578 29267 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195)
I20260807 11:10:48.516454 26041 raft_consensus.cc:2454] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:48.518582 25872 leader_election.cc:302] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:48.519448 29609 raft_consensus.cc:2781] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:10:48.274577 25921 rpcz_store.cc:276] 0807 11:10:47.263338 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:47.811102 (+547764us) service_pool.cc:227] Handling call
0807 11:10:47.826717 (+ 15615us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:48.058979 (+232262us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 11:10:48.059398 (+ 419us) log.cc:842] Serialized 14 byte log entry
0807 11:10:48.263832 (+204434us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:48.263888 (+ 56us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:48.266904 (+ 3016us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:48.266961 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:48.267055 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":143525,"prepare.queue_time_us":1679,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":168846,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":4036,"thread_start_us":203942,"threads_started":1}
W20260807 11:10:48.508945 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 150115us
I20260807 11:10:48.523387 25750 leader_election.cc:302] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:48.477504 29445 consensus_queue.cc:1035] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
W20260807 11:10:48.530512 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 49405.7us
I20260807 11:10:48.532716 26033 raft_consensus.cc:2454] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 7.
I20260807 11:10:48.491387 29516 raft_consensus.cc:3037] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:48.535939 25872 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:48.492110 29380 raft_consensus.cc:686] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:48.547549 25752 leader_election.cc:302] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:48.550196 25869 leader_election.cc:302] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:48.510033 29572 raft_consensus.cc:2726] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:48.534539 29342 raft_consensus.cc:2781] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260807 11:10:48.557581 29601 log.cc:924] Time spent T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260807 11:10:48.546229 28964 raft_consensus.cc:2726] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260807 11:10:48.587451 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 51629.6us
I20260807 11:10:48.588828 26936 leader_election.cc:302] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:48.568667 29342 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:48.529176 29608 consensus_queue.cc:1035] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.068s
I20260807 11:10:48.595739 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:48.596257 29254 raft_consensus.cc:2781] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader election won for term 7
I20260807 11:10:48.596508 25920 raft_consensus.cc:2454] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:10:48.596863 29254 raft_consensus.cc:686] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 7 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:48.597865 29254 consensus_queue.cc:237] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:48.510665 29394 consensus_queue.cc:237] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:48.601547 25921 raft_consensus.cc:1261] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260807 11:10:48.602322 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 84637.7us
I20260807 11:10:48.559577 29330 consensus_queue.cc:1035] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20260807 11:10:48.612073 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:48.621520 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 74052us
I20260807 11:10:48.601835 29342 raft_consensus.cc:3037] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
W20260807 11:10:48.630906 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 54188.3us
I20260807 11:10:48.623955 29205 raft_consensus.cc:482] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:48.634680 29630 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:48.634994 29635 consensus_queue.cc:1035] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260807 11:10:48.637248 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 48501.7us
I20260807 11:10:48.647430 26936 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:48.637846 29417 raft_consensus.cc:2781] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260807 11:10:48.648554 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 98480.7us
I20260807 11:10:48.649801 25869 leader_election.cc:302] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:48.645058 29609 raft_consensus.cc:686] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 5 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:48.659001 29558 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:48.602087 28964 consensus_queue.cc:1035] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.044s
I20260807 11:10:48.664079 25752 leader_election.cc:302] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:48.665050 26936 leader_election.cc:302] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:48.644752 29173 raft_consensus.cc:2726] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:48.668581 25869 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:48.669793 29634 raft_consensus.cc:2726] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:48.670383 29278 raft_consensus.cc:2726] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:48.674232 29510 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:48.674670 29624 raft_consensus.cc:2726] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:48.648902 29417 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:48.673799 25750 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:48.648916 29205 raft_consensus.cc:504] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:48.683274 25869 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:48.662284 29631 raft_consensus.cc:2726] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 11:10:48.675284 29415 consensus_queue.cc:1035] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:10:48.684058 26936 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:48.687044 25752 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:48.667516 28964 consensus_queue.cc:1225] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer 4fd6381544574ddca559ce63b9243efc log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260807 11:10:48.696074 29631 raft_consensus.cc:2726] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 11:10:48.682989 29417 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:10:48.698407 25869 leader_election.cc:302] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:48.719694 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 87353.8us
I20260807 11:10:48.722235 25806 raft_consensus.cc:1226] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 4. Current term is 5. Ops: []
W20260807 11:10:48.733393 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 49982.5us
I20260807 11:10:48.735343 29261 raft_consensus.cc:2726] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 11:10:48.735692 25869 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:48.736815 29261 raft_consensus.cc:3037] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:48.688879 29593 consensus_queue.cc:1046] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260807 11:10:48.737905 25869 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:48.738245 29593 consensus_queue.cc:1035] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 11:10:48.738656 29634 raft_consensus.cc:2781] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:10:48.739027 29634 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:48.739373 29634 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:48.656965 29380 consensus_queue.cc:237] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:48.687808 29173 raft_consensus.cc:2726] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:48.742825 25869 leader_election.cc:302] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:48.743870 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 51554.4us
I20260807 11:10:48.744117 26042 raft_consensus.cc:1226] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260807 11:10:48.722481 29621 raft_consensus.cc:2781] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:48.722394 29217 raft_consensus.cc:3032] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:48.658869 29630 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:48.750916 25869 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:48.695286 29624 raft_consensus.cc:2741] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260807 11:10:48.733078 29608 raft_consensus.cc:3037] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:48.689671 29558 consensus_queue.cc:1035] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
I20260807 11:10:48.776868 29636 consensus_queue.cc:1046] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:10:48.777436 29205 leader_election.cc:288] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:48.759132 29623 raft_consensus.cc:2726] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260807 11:10:48.773823 29615 log.cc:924] Time spent T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.102s user 0.000s sys 0.001s
I20260807 11:10:48.759202 29644 raft_consensus.cc:3037] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:48.760821 29621 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:48.761910 29217 raft_consensus.cc:729] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:48.812493 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
W20260807 11:10:48.818830 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 57293.3us
I20260807 11:10:48.820127 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:48.820837 25806 raft_consensus.cc:2454] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:48.821707 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
W20260807 11:10:48.822328 29625 log.cc:924] Time spent T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.150s user 0.003s sys 0.000s
I20260807 11:10:48.822801 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:48.823328 25806 raft_consensus.cc:3037] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
W20260807 11:10:48.824234 29500 consensus_meta.cc:287] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.546s user 0.004s sys 0.005s
I20260807 11:10:48.814021 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:48.827349 25869 leader_election.cc:302] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:48.827898 25801 raft_consensus.cc:2396] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 11:10:48.813110 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:48.812348 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:48.809975 29169 consensus_meta.cc:287] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.563s user 0.006s sys 0.006s
I20260807 11:10:48.816815 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:48.818020 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:48.800987 29589 consensus_meta.cc:287] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.639s user 0.008s sys 0.005s
I20260807 11:10:48.825101 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:48.829141 25798 raft_consensus.cc:2396] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20260807 11:10:48.815860 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 141424us
I20260807 11:10:48.759549 29254 raft_consensus.cc:482] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:48.801247 29633 raft_consensus.cc:2781] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:48.838099 29633 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:48.838609 29633 raft_consensus.cc:3037] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:48.838922 29630 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:48.839602 29254 raft_consensus.cc:504] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:48.843557 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:48.844354 25917 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:48.847025 29254 leader_election.cc:288] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:48.848196 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:48.848980 26033 raft_consensus.cc:2454] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:48.850952 25869 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:48.852465 25869 leader_election.cc:302] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:48.854653 25872 leader_election.cc:302] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:48.860548 25869 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:48.862083 29254 raft_consensus.cc:2726] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:10:48.826382 29621 raft_consensus.cc:3037] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:48.868799 29254 raft_consensus.cc:2726] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:48.869684 29254 raft_consensus.cc:2781] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:48.870151 29254 raft_consensus.cc:482] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:48.870571 29254 raft_consensus.cc:3037] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:48.759250 29631 consensus_queue.cc:1035] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.076s
W20260807 11:10:48.832119 29569 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.004s sys 0.013s
W20260807 11:10:48.874032 29569 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s user 0.004s sys 0.013s
I20260807 11:10:48.875308 29569 raft_consensus.cc:2726] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:10:48.880285 29621 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:48.883430 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:48.884008 25917 raft_consensus.cc:3037] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:48.834573 29636 raft_consensus.cc:2781] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:48.889930 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:48.890583 25798 raft_consensus.cc:1226] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 5. Current term is 6. Ops: []
I20260807 11:10:48.890909 26033 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:48.776901 29493 consensus_queue.cc:1046] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.088s
I20260807 11:10:48.889991 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:48.889887 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:48.862633 28964 consensus_queue.cc:1035] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260807 11:10:48.833896 29415 consensus_queue.cc:1046] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.058s
I20260807 11:10:48.835346 29500 raft_consensus.cc:504] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:48.848866 29169 raft_consensus.cc:504] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:48.893229 29510 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:48.926605 29642 log.cc:924] Time spent T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.157s user 0.000s sys 0.001s
I20260807 11:10:48.927639 29169 leader_election.cc:288] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:48.937013 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 77260.6us
I20260807 11:10:48.938162 25981 leader_election.cc:302] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:48.939091 25981 leader_election.cc:302] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:48.939926 25981 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:48.901408 25797 raft_consensus.cc:1261] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20260807 11:10:48.890997 29636 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:48.946350 25800 raft_consensus.cc:2454] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:48.834151 29394 consensus_queue.cc:1046] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.061s
W20260807 11:10:48.962656 29629 log.cc:924] Time spent T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.339s user 0.002s sys 0.000s
I20260807 11:10:48.963867 29621 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195)
I20260807 11:10:48.793303 29609 consensus_queue.cc:237] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:48.966293 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:48.885321 29635 consensus_queue.cc:1035] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260807 11:10:48.856112 29589 raft_consensus.cc:504] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:48.886292 29342 raft_consensus.cc:504] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:48.984614 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:48.895957 29631 consensus_queue.cc:1035] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.073s
W20260807 11:10:48.979223 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 77384.8us
I20260807 11:10:48.994464 26033 raft_consensus.cc:3037] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:48.966149 29636 raft_consensus.cc:3037] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
W20260807 11:10:48.984822 29607 log.cc:924] Time spent T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.081s user 0.000s sys 0.001s
W20260807 11:10:49.014079 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 74228.9us
I20260807 11:10:48.867936 29624 consensus_queue.cc:1035] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.128s
I20260807 11:10:49.043512 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:49.044633 26974 raft_consensus.cc:2454] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:49.047433 26041 raft_consensus.cc:1261] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20260807 11:10:49.049535 29330 consensus_queue.cc:1035] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:10:49.056406 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:49.104928 29254 raft_consensus.cc:504] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:49.009604 29394 raft_consensus.cc:482] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
W20260807 11:10:49.132256 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 71470.3us
I20260807 11:10:48.993081 29510 consensus_queue.cc:1035] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.117s
I20260807 11:10:48.964669 29639 consensus_queue.cc:1046] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.147s
I20260807 11:10:49.118733 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:49.122473 29493 raft_consensus.cc:2781] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260807 11:10:49.131054 29598 log.cc:924] Time spent T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.612s user 0.004s sys 0.001s
I20260807 11:10:49.135330 29608 raft_consensus.cc:2726] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:49.182796 29493 raft_consensus.cc:482] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
W20260807 11:10:49.193853 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 191373us
I20260807 11:10:49.194597 29493 raft_consensus.cc:3037] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:49.197683 26033 raft_consensus.cc:2454] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:49.199198 29500 leader_election.cc:288] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:49.200520 29589 leader_election.cc:288] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:49.202822 25921 raft_consensus.cc:1226] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260807 11:10:49.205157 29493 raft_consensus.cc:504] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:49.213752 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 44449.6us
I20260807 11:10:49.218264 25872 leader_election.cc:302] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:49.182956 29678 raft_consensus.cc:2781] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:49.183851 26975 raft_consensus.cc:2454] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:49.133543 29394 raft_consensus.cc:504] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:49.213152 29597 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.700s user 0.007s sys 0.008s
I20260807 11:10:49.259939 25981 leader_election.cc:302] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:10:49.221417 29515 consensus_meta.cc:287] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 1.043s user 0.011s sys 0.002s
W20260807 11:10:49.267243 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 406783us
I20260807 11:10:49.209739 29670 raft_consensus.cc:2781] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:49.270171 29670 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:49.272241 29670 raft_consensus.cc:3037] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:49.227136 29678 raft_consensus.cc:482] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
W20260807 11:10:49.276721 29665 log.cc:924] Time spent T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.284s user 0.000s sys 0.001s
I20260807 11:10:49.277840 29515 consensus_queue.cc:1035] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260807 11:10:49.278302 29659 log.cc:924] Time spent T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.109s user 0.000s sys 0.001s
W20260807 11:10:49.278494 29658 log.cc:924] Time spent T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.263s user 0.000s sys 0.001s
I20260807 11:10:49.279286 29515 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:49.279950 29254 leader_election.cc:288] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:49.279811 29515 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:49.281536 29037 consensus_meta.cc:287] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.813s user 0.010s sys 0.001s
I20260807 11:10:49.282006 29037 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:49.283999 29037 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:49.283691 29633 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:49.285740 29515 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:49.286339 29515 raft_consensus.cc:2741] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260807 11:10:49.289705 29493 leader_election.cc:288] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:49.293063 29342 leader_election.cc:288] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:49.268694 29417 consensus_meta.cc:287] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.531s user 0.007s sys 0.003s
W20260807 11:10:49.265705 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 400910us
W20260807 11:10:49.307610 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 105611us
I20260807 11:10:49.268626 25869 leader_election.cc:302] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:49.298166 29394 leader_election.cc:288] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:49.263361 29597 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.750s user 0.014s sys 0.022s
W20260807 11:10:49.298202 29516 consensus_meta.cc:287] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.738s user 0.011s sys 0.000s
W20260807 11:10:49.299799 29261 consensus_meta.cc:287] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.562s user 0.009s sys 0.000s
I20260807 11:10:49.303504 29267 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:49.304611 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260807 11:10:49.313980 29634 consensus_meta.cc:287] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.574s user 0.007s sys 0.006s
I20260807 11:10:49.306360 25917 raft_consensus.cc:2454] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:10:49.224105 29608 raft_consensus.cc:2781] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:49.372610 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:49.397760 26033 raft_consensus.cc:3037] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:49.399433 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:49.400061 26046 raft_consensus.cc:3037] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
W20260807 11:10:49.401059 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 141234us
W20260807 11:10:49.431490 25750 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.005s sys 0.010s
W20260807 11:10:49.446655 25750 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.561s user 0.006s sys 0.011s
W20260807 11:10:49.447105 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 586805us
I20260807 11:10:49.355773 29445 raft_consensus.cc:3037] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:49.356293 29597 raft_consensus.cc:686] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:49.485409 26033 raft_consensus.cc:2454] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:49.394809 29261 raft_consensus.cc:2726] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:49.371572 29267 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:49.366101 29307 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.848s user 0.000s sys 0.010s
W20260807 11:10:49.485747 25806 consensus_meta.cc:287] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.662s user 0.013s sys 0.001s
I20260807 11:10:49.541333 25806 raft_consensus.cc:2454] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:49.275380 29678 raft_consensus.cc:3037] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:49.307140 29417 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:49.497205 29013 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.592s user 0.006s sys 0.011s
I20260807 11:10:49.448400 29608 raft_consensus.cc:686] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
W20260807 11:10:49.556406 29307 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.038s user 0.001s sys 0.010s
W20260807 11:10:49.561682 29665 log.cc:924] Time spent T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.118s user 0.001s sys 0.000s
I20260807 11:10:49.376741 29516 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:49.635070 29516 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:49.636925 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 368425us
I20260807 11:10:49.404188 29634 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:49.642499 25752 leader_election.cc:302] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:49.638163 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 156844us
I20260807 11:10:49.659775 25872 leader_election.cc:302] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:49.518741 29624 consensus_queue.cc:1035] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.120s
W20260807 11:10:49.731801 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 89413us
I20260807 11:10:49.663566 29173 raft_consensus.cc:3032] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 5 LEADER]: Stepping down as leader of term 5
I20260807 11:10:50.377974 29593 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:50.426414 29173 raft_consensus.cc:729] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:50.426663 29593 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:49.740418 29597 consensus_queue.cc:237] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:50.427652 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:10:50.137925 29710 log.cc:924] Time spent T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.163s user 0.000s sys 0.000s
I20260807 11:10:49.755925 29415 consensus_queue.cc:1035] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.684s
I20260807 11:10:50.429764 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1691) took 1376 ms. Trace:
I20260807 11:10:50.429659 29415 consensus_queue.cc:1046] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:10:50.430595 29415 consensus_queue.cc:1225] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer b22d69db45194908ad3d345eba9badd9 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260807 11:10:50.431354 29415 raft_consensus.cc:3032] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 11:10:50.430323 26041 rpcz_store.cc:276] 0807 11:10:49.053290 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:49.053638 (+ 348us) service_pool.cc:227] Handling call
0807 11:10:49.054085 (+ 447us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:49.125951 (+ 71866us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:49.126034 (+ 83us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:49.172463 (+ 46429us) log.cc:842] Serialized 26 byte log entry
0807 11:10:49.360669 (+188206us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:49.360731 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:49.360839 (+ 108us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:50.429557 (+1068718us) raft_consensus.cc:1671] finished
0807 11:10:50.429577 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:50.429712 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":64660,"raft.queue_time_us":221749,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":125,"thread_start_us":143900,"threads_started":1,"wal-append.queue_time_us":539947}
I20260807 11:10:50.431716 29415 raft_consensus.cc:729] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
W20260807 11:10:49.651404 29013 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.689s user 0.010s sys 0.011s
I20260807 11:10:50.049371 29342 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:49.745216 28964 consensus_queue.cc:1035] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.683s
W20260807 11:10:50.425379 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 765503us
I20260807 11:10:49.703790 25803 raft_consensus.cc:1226] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:50.434398 29029 consensus_queue.cc:1035] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:10:50.494560 25869 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:50.426577 25752 leader_election.cc:302] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:50.447800 29510 raft_consensus.cc:2932] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
W20260807 11:10:49.703877 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 229252us
I20260807 11:10:49.832341 29608 consensus_queue.cc:237] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:49.777688 29669 raft_consensus.cc:2781] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:49.663821 29633 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:50.443549 25797 raft_consensus.cc:1261] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260807 11:10:50.250550 29609 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.613s user 0.009s sys 0.012s
W20260807 11:10:50.408198 26938 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.664s user 0.012s sys 0.016s
I20260807 11:10:49.706283 29516 raft_consensus.cc:2726] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:49.911794 29697 consensus_queue.cc:1046] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.749s
I20260807 11:10:50.473802 26936 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:49.673638 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:50.189152 29678 consensus_meta.cc:287] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.607s user 0.010s sys 0.001s
I20260807 11:10:50.449354 29558 raft_consensus.cc:3032] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Stepping down as leader of term 3
W20260807 11:10:50.455164 26046 consensus_meta.cc:287] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 1.055s user 0.006s sys 0.007s
I20260807 11:10:50.444585 29711 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:49.814066 29682 raft_consensus.cc:3032] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 LEADER]: Stepping down as leader of term 7
W20260807 11:10:50.156919 29683 log.cc:924] Time spent T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.424s user 0.001s sys 0.001s
I20260807 11:10:50.465236 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:10:50.052657 29663 log.cc:924] Time spent T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.512s user 0.000s sys 0.007s
I20260807 11:10:50.283070 29705 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
W20260807 11:10:50.138957 29666 log.cc:924] Time spent T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.164s user 0.000s sys 0.000s
I20260807 11:10:50.138177 29692 consensus_queue.cc:1046] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.505s
W20260807 11:10:50.366961 25800 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.674s user 0.001s sys 0.000s
W20260807 11:10:50.534153 25800 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.841s user 0.003s sys 0.001s
I20260807 11:10:50.535156 25800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 331) took 1354 ms. Trace:
W20260807 11:10:50.148273 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 60293.4us
I20260807 11:10:50.168807 29261 raft_consensus.cc:482] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:50.160327 29636 consensus_meta.cc:287] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 1.046s user 0.012s sys 0.000s
I20260807 11:10:50.447041 26041 raft_consensus.cc:1226] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:50.440294 29708 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:50.439069 29691 raft_consensus.cc:482] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:50.496937 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
W20260807 11:10:50.415161 29644 consensus_meta.cc:287] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 1.585s user 0.006s sys 0.008s
I20260807 11:10:50.074687 29698 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:49.912977 29634 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:50.511502 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 85063.6us
W20260807 11:10:50.513516 26938 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.773s user 0.017s sys 0.016s
I20260807 11:10:50.513641 28964 raft_consensus.cc:2726] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:50.551021 25752 leader_election.cc:302] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
W20260807 11:10:50.524658 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 50996.7us
I20260807 11:10:50.516373 26046 raft_consensus.cc:2454] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:10:50.515532 25981 leader_election.cc:302] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:10:50.513080 29609 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.876s user 0.013s sys 0.021s
I20260807 11:10:50.512331 29633 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:50.516566 29711 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:50.502954 29342 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:50.516175 29558 raft_consensus.cc:729] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
W20260807 11:10:50.490193 29649 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.873s user 0.018s sys 0.018s
I20260807 11:10:50.543159 25801 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:10:50.523425 29417 leader_election.cc:288] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 8 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:50.517148 26033 raft_consensus.cc:3037] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:50.514264 29697 raft_consensus.cc:482] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:50.516721 29682 raft_consensus.cc:729] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:50.550984 26938 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:50.527153 29693 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:50.576931 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59312 (request call id 1693) took 1196 ms. Trace:
I20260807 11:10:50.626081 26046 rpcz_store.cc:276] 0807 11:10:49.363577 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:49.365724 (+ 2147us) service_pool.cc:227] Handling call
0807 11:10:50.560072 (+1194348us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:50.579828 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:50.627259 26041 raft_consensus.cc:2454] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:10:50.602034 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:10:50.593458 26937 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.642s user 0.007s sys 0.002s
I20260807 11:10:50.516003 29678 raft_consensus.cc:504] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:50.602603 29649 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.996s user 0.018s sys 0.018s
I20260807 11:10:50.603143 29634 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:50.557986 29597 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:50.634176 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:50.635485 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:50.655000 25917 raft_consensus.cc:3037] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:50.554569 29669 raft_consensus.cc:686] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:50.546516 29307 consensus_queue.cc:1035] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
I20260807 11:10:50.656438 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:50.657111 25796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 325) took 1968 ms. Trace:
W20260807 11:10:50.640442 29683 log.cc:924] Time spent T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260807 11:10:50.549536 29719 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:50.638135 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 87234.9us
I20260807 11:10:50.558070 29644 raft_consensus.cc:2726] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:50.636623 26033 raft_consensus.cc:2454] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 7.
I20260807 11:10:50.642971 29609 raft_consensus.cc:482] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:50.554479 29608 raft_consensus.cc:2781] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election won for term 4
W20260807 11:10:50.641983 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 147553us
I20260807 11:10:50.642226 26980 raft_consensus.cc:1261] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260807 11:10:50.650866 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:50.615082 29624 raft_consensus.cc:2781] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:50.557756 29261 raft_consensus.cc:504] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:50.651451 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:50.602576 29633 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:50.637120 25931 raft_consensus.cc:1261] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260807 11:10:50.633561 29693 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:50.657618 25796 rpcz_store.cc:276] 0807 11:10:48.689042 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:48.689183 (+ 141us) service_pool.cc:227] Handling call
0807 11:10:48.717135 (+ 27952us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:48.749437 (+ 32302us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:48.750259 (+ 822us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:48.761591 (+ 11332us) log.cc:842] Serialized 26 byte log entry
0807 11:10:48.894563 (+132972us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:10:48.894601 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:48.894664 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:50.656890 (+1762226us) raft_consensus.cc:1671] finished
0807 11:10:50.656914 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:50.657056 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26605,"raft.queue_time_us":32360,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":1710,"thread_start_us":116352,"threads_started":1,"wal-append.queue_time_us":586710}
I20260807 11:10:50.542985 29700 consensus_queue.cc:1046] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.119s
I20260807 11:10:50.609302 29029 raft_consensus.cc:2781] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:50.675550 25872 leader_election.cc:302] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:50.676347 25932 raft_consensus.cc:1226] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: [2.4-4.5]
I20260807 11:10:50.650697 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:50.677419 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:50.641044 29417 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:50.634979 29697 raft_consensus.cc:504] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:50.546653 29636 raft_consensus.cc:504] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:50.639465 26937 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.899s user 0.008s sys 0.002s
I20260807 11:10:50.640321 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:50.605144 29692 consensus_queue.cc:1046] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.067s
I20260807 11:10:50.690109 25925 raft_consensus.cc:2379] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 7: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:50.692209 25919 raft_consensus.cc:2379] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 7: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:50.691236 26938 leader_election.cc:302] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:50.533615 29705 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:50.683693 26974 raft_consensus.cc:1261] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260807 11:10:50.557622 29691 raft_consensus.cc:504] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:50.699458 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:50.700664 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:50.707427 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:50.707479 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:50.637467 29415 consensus_queue.cc:260] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:50.784982 25920 raft_consensus.cc:2379] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 6: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:50.732016 25918 raft_consensus.cc:2454] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:50.578199 29698 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:50.736268 25916 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 273) took 1994 ms. Trace:
I20260807 11:10:50.742002 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:50.692693 29624 raft_consensus.cc:482] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:50.788640 25932 raft_consensus.cc:2379] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 6: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:50.671710 29634 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:50.550271 29708 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:50.783428 29670 consensus_meta.cc:287] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 1.511s user 0.012s sys 0.003s
I20260807 11:10:50.728983 29700 raft_consensus.cc:3032] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:50.682701 29597 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:50.786410 25916 rpcz_store.cc:276] 0807 11:10:48.742210 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:48.742380 (+ 170us) service_pool.cc:227] Handling call
0807 11:10:48.749813 (+ 7433us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:48.969025 (+219212us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:48.969080 (+ 55us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:48.981328 (+ 12248us) log.cc:842] Serialized 26 byte log entry
0807 11:10:49.018236 (+ 36908us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:10:49.018577 (+ 341us) log.cc:842] Serialized 14 byte log entry
0807 11:10:49.022663 (+ 4086us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:49.022786 (+ 123us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:50.736063 (+1713277us) raft_consensus.cc:1671] finished
0807 11:10:50.736085 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:50.736211 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3712,"prepare.queue_time_us":172,"prepare.run_cpu_time_us":8477,"prepare.run_wall_time_us":57012,"raft.queue_time_us":21890,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":22137,"spinlock_wait_cycles":87720704,"thread_start_us":222262,"threads_started":2,"wal-append.queue_time_us":589004}
I20260807 11:10:50.773499 25917 raft_consensus.cc:2454] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 7.
I20260807 11:10:50.765292 29662 raft_consensus.cc:2781] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:10:50.795415 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
W20260807 11:10:50.791862 29740 log.cc:924] Time spent T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.111s user 0.001s sys 0.000s
I20260807 11:10:50.688838 29609 raft_consensus.cc:504] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:50.698000 29644 consensus_queue.cc:1035] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.091s
I20260807 11:10:50.726558 29029 raft_consensus.cc:482] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:50.798012 26042 raft_consensus.cc:1261] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:10:50.791570 29624 raft_consensus.cc:3037] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:50.828882 29608 raft_consensus.cc:686] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:50.830154 29608 consensus_queue.cc:237] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:50.800159 25932 raft_consensus.cc:1226] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [term 7 LEADER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260807 11:10:50.822691 25799 raft_consensus.cc:2379] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 6: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
W20260807 11:10:50.823613 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 254892us
I20260807 11:10:50.690267 29558 consensus_queue.cc:260] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:50.684326 29719 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:50.820705 25917 pending_rounds.cc:76] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc: Aborting all ops after (but not including) 1
I20260807 11:10:50.796969 29700 raft_consensus.cc:729] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:50.838563 25917 pending_rounds.cc:97] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260807 11:10:50.838904 25917 raft_consensus.cc:2866] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:10:50.826964 25916 raft_consensus.cc:1226] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260807 11:10:50.835834 25752 leader_election.cc:302] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:50.792400 29711 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:50.834280 29667 raft_consensus.cc:2781] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:50.840137 26984 raft_consensus.cc:1261] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260807 11:10:50.760903 29417 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:50.844332 25921 raft_consensus.cc:2454] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:50.839432 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:50.825122 29662 raft_consensus.cc:686] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:50.846899 29634 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:50.844339 29667 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:50.848905 29727 raft_consensus.cc:482] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:50.849886 25801 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 8.
I20260807 11:10:50.821722 28964 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:50.835347 29415 raft_consensus.cc:3037] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:50.797991 29692 consensus_queue.cc:1046] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260807 11:10:50.852403 29342 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:50.834977 29029 raft_consensus.cc:3037] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:50.857170 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:50.875865 25916 raft_consensus.cc:3037] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:50.839975 29720 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:50.796703 29670 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:50.864217 29261 leader_election.cc:288] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:50.864799 29755 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:50.730217 29682 consensus_queue.cc:260] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:50.866030 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1694) took 1498 ms. Trace:
I20260807 11:10:50.854166 29558 raft_consensus.cc:3037] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:50.854553 25932 raft_consensus.cc:3037] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:50.821523 29702 consensus_queue.cc:1035] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.252s
I20260807 11:10:50.879285 29725 raft_consensus.cc:2781] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:10:50.879743 29693 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:10:50.880439 29722 log.cc:924] Time spent T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.078s user 0.000s sys 0.001s
I20260807 11:10:50.865514 29636 leader_election.cc:288] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:50.856803 29728 raft_consensus.cc:482] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:50.797943 29669 consensus_queue.cc:237] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:50.860534 29732 consensus_queue.cc:1035] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.271s
I20260807 11:10:50.880568 29719 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:51.040980 29763 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:51.045583 29719 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:51.046227 29719 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:51.031672 29757 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:51.048026 29755 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:51.048545 29757 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:50.998445 29662 consensus_queue.cc:237] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:51.030791 29714 log.cc:924] Time spent T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.255s user 0.000s sys 0.001s
W20260807 11:10:51.031068 29717 log.cc:924] Time spent T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.310s user 0.001s sys 0.000s
I20260807 11:10:51.054934 29697 leader_election.cc:288] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:51.032585 29748 raft_consensus.cc:482] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:51.056176 29725 raft_consensus.cc:686] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:51.056388 29748 raft_consensus.cc:504] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:51.057361 29725 consensus_queue.cc:237] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:51.059157 29748 leader_election.cc:288] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:51.059875 29748 consensus_queue.cc:1035] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:10:51.061826 29682 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:10:51.042294 29760 raft_consensus.cc:2726] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:10:51.071372 29708 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:51.072839 29609 leader_election.cc:288] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:51.046064 29763 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:51.041016 29745 raft_consensus.cc:482] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:50.860320 29667 raft_consensus.cc:3037] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
W20260807 11:10:50.888422 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 212962us
W20260807 11:10:50.989465 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 137629us
I20260807 11:10:51.146275 29755 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:51.146584 25916 raft_consensus.cc:2454] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
W20260807 11:10:51.006474 29445 consensus_meta.cc:287] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 1.486s user 0.015s sys 0.000s
I20260807 11:10:51.151209 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:51.159528 26033 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:51.051631 29757 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:51.161198 25752 leader_election.cc:302] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:51.161522 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:51.160414 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:51.164419 26033 raft_consensus.cc:2454] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:51.168067 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:51.105966 29732 consensus_queue.cc:1225] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer b22d69db45194908ad3d345eba9badd9 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260807 11:10:50.861847 29678 leader_election.cc:288] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:51.157119 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:50.998781 29633 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:51.171494 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 480339us
I20260807 11:10:51.171795 25872 leader_election.cc:302] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:51.171878 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:51.014811 29644 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:50.875193 28964 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:51.041864 29691 leader_election.cc:288] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:51.043810 29698 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:51.018577 29705 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:50.866880 29727 raft_consensus.cc:504] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:51.108969 29728 raft_consensus.cc:504] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:51.106218 26038 rpcz_store.cc:276] 0807 11:10:49.367459 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:49.368519 (+ 1060us) service_pool.cc:227] Handling call
0807 11:10:49.398668 (+ 30149us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:49.399069 (+ 401us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:49.399097 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:50.477852 (+1078755us) log.cc:842] Serialized 26 byte log entry
0807 11:10:50.655293 (+177441us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:50.655349 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:50.655435 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:50.865811 (+210376us) raft_consensus.cc:1671] finished
0807 11:10:50.865831 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:50.865972 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2318,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":62741,"thread_start_us":120909,"threads_started":1,"wal-append.queue_time_us":291470}
I20260807 11:10:51.160605 29445 raft_consensus.cc:2726] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260807 11:10:51.175485 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 448102us
I20260807 11:10:51.154333 29720 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:51.266389 26938 leader_election.cc:302] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:51.169582 29745 raft_consensus.cc:504] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:50.883807 29700 consensus_queue.cc:260] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:51.177778 26046 raft_consensus.cc:2454] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:51.212960 26041 raft_consensus.cc:1226] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260807 11:10:51.174801 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:10:51.342623 25750 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.004s sys 0.025s
W20260807 11:10:52.038039 25750 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.246s user 0.013s sys 0.040s
W20260807 11:10:52.035095 25916 consensus_meta.cc:287] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.873s user 0.008s sys 0.004s
I20260807 11:10:52.038475 25916 raft_consensus.cc:2454] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:52.039186 25916 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1731) took 1191 ms. Trace:
I20260807 11:10:52.039186 26936 leader_election.cc:302] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:52.044881 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1710) took 1417 ms. Trace:
W20260807 11:10:52.044710 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 872993us
W20260807 11:10:51.338964 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 177825us
W20260807 11:10:51.488122 25981 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.904s user 0.047s sys 0.098s
I20260807 11:10:52.039503 25916 rpcz_store.cc:276] 0807 11:10:50.847771 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:51.161291 (+313520us) service_pool.cc:227] Handling call
0807 11:10:52.039117 (+877826us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:52.056170 25921 rpcz_store.cc:276] 0807 11:10:50.627171 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:50.633920 (+ 6749us) service_pool.cc:227] Handling call
0807 11:10:52.044639 (+1410719us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b54003fedd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:52.044787 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":436189952}
W20260807 11:10:52.058005 25981 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.479s user 0.056s sys 0.116s
I20260807 11:10:52.060056 26033 raft_consensus.cc:1261] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260807 11:10:52.064287 25932 consensus_meta.cc:287] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.920s user 0.010s sys 0.002s
I20260807 11:10:52.064898 25932 raft_consensus.cc:2454] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:52.065706 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1728) took 1245 ms. Trace:
I20260807 11:10:52.066169 25932 rpcz_store.cc:276] 0807 11:10:50.820512 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:50.839119 (+ 18607us) service_pool.cc:227] Handling call
0807 11:10:52.065640 (+1226521us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:51.574251 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:51.161008 29261 raft_consensus.cc:482] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
W20260807 11:10:51.579486 29682 consensus_meta.cc:287] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.517s user 0.007s sys 0.000s
I20260807 11:10:52.068930 25919 raft_consensus.cc:1261] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260807 11:10:52.068578 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:51.867102 29278 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:52.063596 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 1.54807e+06us
I20260807 11:10:51.474092 29636 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:52.051661 29756 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.978s user 0.001s sys 0.010s
I20260807 11:10:52.078485 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:52.044998 29779 raft_consensus.cc:2781] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260807 11:10:52.043514 29558 consensus_meta.cc:287] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.894s user 0.005s sys 0.006s
W20260807 11:10:52.060377 29510 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.003s sys 0.007s
I20260807 11:10:51.357990 29342 raft_consensus.cc:2781] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:10:51.558748 29768 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:52.037490 26034 raft_consensus.cc:3037] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 6
W20260807 11:10:52.090279 29634 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.120s user 0.015s sys 0.019s
I20260807 11:10:51.567360 29702 raft_consensus.cc:3032] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:51.574765 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:51.174970 29597 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:52.085681 25921 raft_consensus.cc:1226] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260807 11:10:52.081387 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:52.094135 29779 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:52.081974 26033 raft_consensus.cc:2379] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 5: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:51.514624 29697 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:51.521394 29692 raft_consensus.cc:2781] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:51.560974 29764 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:51.471932 29763 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:51.797855 29624 consensus_meta.cc:287] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.962s user 0.009s sys 0.003s
I20260807 11:10:52.145903 29624 raft_consensus.cc:504] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:52.087268 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:52.149322 26041 raft_consensus.cc:2379] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 6: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:51.585798 29417 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:52.045105 29415 consensus_meta.cc:287] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 1.170s user 0.012s sys 0.000s
I20260807 11:10:52.080919 26975 raft_consensus.cc:3037] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:51.558252 29719 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:52.058109 29644 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:52.043681 29728 leader_election.cc:288] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:52.089421 29278 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:52.043701 29727 leader_election.cc:288] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:52.094295 25932 raft_consensus.cc:1261] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 11:10:52.091252 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:52.137132 29702 raft_consensus.cc:729] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
W20260807 11:10:52.095064 25869 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.109s user 0.001s sys 0.022s
W20260807 11:10:52.086895 29608 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.147s user 0.005s sys 0.016s
I20260807 11:10:52.113262 29768 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:52.092675 29667 consensus_meta.cc:287] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.932s user 0.010s sys 0.001s
W20260807 11:10:52.067351 29029 consensus_meta.cc:287] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 1.126s user 0.008s sys 0.004s
W20260807 11:10:52.176360 29608 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.295s user 0.006s sys 0.017s
I20260807 11:10:52.091701 29700 raft_consensus.cc:3037] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
W20260807 11:10:52.138262 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 45697us
I20260807 11:10:52.140065 26038 raft_consensus.cc:2396] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 6 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 11:10:52.177690 29705 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:52.084957 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 45843.9us
I20260807 11:10:52.194388 29415 consensus_queue.cc:1035] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20260807 11:10:52.084396 29261 raft_consensus.cc:504] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:52.174849 29719 raft_consensus.cc:482] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:52.140461 26034 raft_consensus.cc:2454] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:10:52.143745 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1726) took 1414 ms. Trace:
W20260807 11:10:52.101799 29510 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.723s user 0.005s sys 0.007s
I20260807 11:10:52.140328 29778 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:52.114640 29745 leader_election.cc:288] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:52.150329 25806 raft_consensus.cc:1226] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260807 11:10:52.130697 29634 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.161s user 0.018s sys 0.026s
I20260807 11:10:52.151623 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:52.153057 25872 leader_election.cc:302] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:52.139439 29682 raft_consensus.cc:2781] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 11:10:52.129005 28964 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:52.157043 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:52.204010 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1723) took 1517 ms. Trace:
I20260807 11:10:52.157735 29764 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:52.204624 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59312 (request call id 1701) took 1048 ms. Trace:
I20260807 11:10:52.158537 29697 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:52.148517 29624 leader_election.cc:288] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:52.093827 29670 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:52.081060 29775 consensus_queue.cc:1046] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.067s
I20260807 11:10:52.093222 29636 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:51.290558 29516 consensus_queue.cc:1035] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.796s
I20260807 11:10:52.173573 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1727) took 1387 ms. Trace:
W20260807 11:10:52.175524 25869 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.190s user 0.002s sys 0.022s
I20260807 11:10:52.177465 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:52.252091 25932 rpcz_store.cc:276] 0807 11:10:50.786566 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.088711 (+1302145us) service_pool.cc:227] Handling call
0807 11:10:52.090398 (+ 1687us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:52.173234 (+ 82836us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:52.173499 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:52.252524 26046 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:52.150336 29692 raft_consensus.cc:482] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:52.150799 29779 raft_consensus.cc:3037] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
W20260807 11:10:52.176909 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 910557us
I20260807 11:10:52.192571 25981 leader_election.cc:302] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:52.160722 29763 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:52.090515 29756 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.069s user 0.001s sys 0.010s
I20260807 11:10:52.163704 29781 raft_consensus.cc:2781] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 11:10:52.213506 26034 rpcz_store.cc:276] 0807 11:10:51.155999 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:51.156135 (+ 136us) service_pool.cc:227] Handling call
0807 11:10:52.204563 (+1048428us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:52.209688 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:52.197470 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1719) took 1521 ms. Trace:
I20260807 11:10:52.211818 25917 rpcz_store.cc:276] 0807 11:10:50.686235 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:50.820125 (+133890us) service_pool.cc:227] Handling call
0807 11:10:50.820511 (+ 386us) raft_consensus.cc:1457] Updating replica for 3 ops
0807 11:10:51.210622 (+390111us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:51.210657 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0807 11:10:51.211367 (+ 710us) log.cc:842] Serialized 78 byte log entry
0807 11:10:51.814885 (+603518us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:10:51.814935 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:51.815039 (+ 104us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:52.203785 (+388746us) raft_consensus.cc:1671] finished
0807 11:10:52.203810 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:52.203955 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11674,"raft.queue_time_us":118319,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":37784,"thread_start_us":802270,"threads_started":2,"wal-append.queue_time_us":821814}
I20260807 11:10:52.202090 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:52.277448 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:52.277598 26038 raft_consensus.cc:2454] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 7.
I20260807 11:10:52.278021 26046 raft_consensus.cc:3037] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:52.206725 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1686) took 3532 ms. Trace:
W20260807 11:10:52.278540 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 104071us
I20260807 11:10:52.279789 25752 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:52.207595 25806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1709) took 1514 ms. Trace:
I20260807 11:10:52.277274 29798 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:52.281215 25806 rpcz_store.cc:276] 0807 11:10:50.693445 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:50.695141 (+ 1696us) service_pool.cc:227] Handling call
0807 11:10:50.795802 (+100661us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:52.205723 (+1409921us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:52.207349 (+ 1626us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54002d50d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:52.207538 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3105624704}
I20260807 11:10:52.205127 29778 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:52.282166 29798 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:52.210965 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 58027.4us
I20260807 11:10:52.205349 29768 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:52.222046 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:52.284004 29778 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:52.284216 26041 raft_consensus.cc:2379] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 5: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:52.284623 29781 raft_consensus.cc:686] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:52.255293 29692 raft_consensus.cc:3037] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:52.285591 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:52.285657 29781 consensus_queue.cc:237] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:52.286650 26041 raft_consensus.cc:3037] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:52.195948 29029 raft_consensus.cc:504] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:52.289078 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:52.289724 26034 raft_consensus.cc:2454] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:10:52.290525 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:52.290787 29029 leader_election.cc:288] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:52.291091 26034 raft_consensus.cc:2454] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:10:52.291936 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:52.205905 29702 consensus_queue.cc:260] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:52.305318 29702 raft_consensus.cc:3037] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:52.304706 26034 raft_consensus.cc:2454] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:52.219468 29682 raft_consensus.cc:686] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:52.195114 29342 raft_consensus.cc:686] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:52.309680 29794 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:52.310232 29342 consensus_queue.cc:237] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:52.310753 29794 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:52.203487 29667 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:52.314845 29667 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:52.204375 29261 leader_election.cc:288] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:52.203450 26975 raft_consensus.cc:2454] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:52.250391 29777 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:52.250627 29205 raft_consensus.cc:3032] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:52.316447 29205 raft_consensus.cc:729] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:52.316488 29777 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:52.317488 29205 consensus_queue.cc:260] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:52.318550 29205 raft_consensus.cc:3037] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:52.318782 29777 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:52.320487 29794 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:52.253679 25932 raft_consensus.cc:1261] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 11:10:52.321256 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1730) took 1473 ms. Trace:
I20260807 11:10:52.201974 29644 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:52.321754 29267 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:52.213397 29745 raft_consensus.cc:2781] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:52.260185 29173 consensus_queue.cc:260] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:52.264756 29764 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:52.266216 29697 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:52.266800 28964 consensus_queue.cc:1035] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 11:10:52.266901 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:52.204649 25921 rpcz_store.cc:276] 0807 11:10:50.729487 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.075960 (+1346473us) service_pool.cc:227] Handling call
0807 11:10:52.085419 (+ 9459us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:52.143410 (+ 57991us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:52.143663 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:52.202332 29705 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:52.291983 25869 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:52.292414 26033 raft_consensus.cc:2379] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 6: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:52.292690 29728 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:52.293146 29655 log.cc:924] Time spent T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 2.676s user 0.042s sys 0.027s
I20260807 11:10:52.293221 29787 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:52.293462 29727 raft_consensus.cc:482] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:52.276360 29763 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:52.293764 29608 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:52.294198 29670 raft_consensus.cc:2741] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260807 11:10:52.294827 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:52.278716 26039 rpcz_store.cc:276] 0807 11:10:48.673941 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:48.681663 (+ 7722us) service_pool.cc:227] Handling call
0807 11:10:48.693526 (+ 11863us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:48.694017 (+ 491us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:48.694058 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:48.694612 (+ 554us) log.cc:842] Serialized 26 byte log entry
0807 11:10:48.772307 (+ 77695us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:10:48.772347 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:48.772412 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:52.205524 (+3433112us) spinlock_profiling.cc:243] Waited 5.86 ms on lock 0x7b24002c0408. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc5f4cf4 00007f3fbc5f2dfb 00007f3fbc681559 00007f3fbc6811e5 00007f3fbc680740 00007f3fbf373578 00007f3fbf373cd8 00007f3fbf34b6e8 00007f3fbf346aea 00007f3fbf345f36
0807 11:10:52.206446 (+ 922us) raft_consensus.cc:1671] finished
0807 11:10:52.206465 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:52.206667 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2799,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":6185,"spinlock_wait_cycles":13470592,"thread_start_us":49116,"threads_started":1,"wal-append.queue_time_us":245455}
I20260807 11:10:52.296051 26938 leader_election.cc:302] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:52.276127 25925 rpcz_store.cc:276] 0807 11:10:50.675739 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:50.785908 (+110169us) service_pool.cc:227] Handling call
0807 11:10:50.794028 (+ 8120us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:50.794369 (+ 341us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:50.794398 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:50.819545 (+ 25147us) log.cc:842] Serialized 26 byte log entry
0807 11:10:51.290579 (+471034us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:51.290635 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:51.290740 (+ 105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:52.197254 (+906514us) raft_consensus.cc:1671] finished
0807 11:10:52.197277 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:52.197414 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31084,"raft.queue_time_us":68756,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":66143,"thread_start_us":455850,"threads_started":1,"wal-append.queue_time_us":1302342}
I20260807 11:10:52.277297 25917 raft_consensus.cc:1226] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: []
W20260807 11:10:52.343813 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 47850.4us
I20260807 11:10:52.344430 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1732) took 1496 ms. Trace:
I20260807 11:10:52.344614 26938 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:52.344821 25917 rpcz_store.cc:276] 0807 11:10:50.848057 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.276740 (+1428683us) service_pool.cc:227] Handling call
0807 11:10:52.277104 (+ 364us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:52.344184 (+ 67080us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:52.344385 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:52.284214 29798 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:52.207065 29720 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:52.351764 29267 raft_consensus.cc:2781] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:10:52.352258 29267 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:52.352656 29267 raft_consensus.cc:3037] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:52.353216 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:52.353533 29593 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:52.353979 26038 raft_consensus.cc:2354] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4.
I20260807 11:10:52.354779 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:52.354431 29682 consensus_queue.cc:237] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:52.355376 26038 raft_consensus.cc:3037] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:52.268946 29415 raft_consensus.cc:482] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:52.204106 29719 raft_consensus.cc:504] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:52.357033 29415 raft_consensus.cc:504] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:52.357944 29705 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:52.306835 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:52.347030 29608 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:52.358280 29719 leader_election.cc:288] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:52.359279 29415 leader_election.cc:288] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:52.321564 25932 rpcz_store.cc:276] 0807 11:10:50.847410 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.252988 (+1405578us) service_pool.cc:227] Handling call
0807 11:10:52.253279 (+ 291us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:52.320977 (+ 67698us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:52.321203 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:52.248885 29636 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:52.341732 29728 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:52.362318 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:52.362776 29728 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:52.363324 29778 raft_consensus.cc:2781] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:52.363771 29778 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:52.364219 29778 raft_consensus.cc:3037] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:52.343134 29727 raft_consensus.cc:504] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:52.367653 29727 leader_election.cc:288] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:52.367861 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:52.368599 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:52.369104 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:52.369285 25798 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:52.331717 29692 raft_consensus.cc:504] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:52.370654 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:52.337368 29173 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:52.372102 25803 raft_consensus.cc:1226] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:52.372381 29692 leader_election.cc:288] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:52.338253 26035 raft_consensus.cc:2396] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260807 11:10:52.360468 29608 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:52.338027 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 136800us
I20260807 11:10:52.339051 25981 leader_election.cc:398] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 election: Vote denied by peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465) with higher term. Message: Invalid argument: T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3.
I20260807 11:10:52.328814 29745 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 11:10:52.341727 26979 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 455) took 1780 ms (client timeout 1906 ms). Trace:
W20260807 11:10:52.380802 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 97442.7us
W20260807 11:10:52.390568 26979 rpcz_store.cc:269] 0807 11:10:50.561110 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:50.568017 (+ 6907us) service_pool.cc:227] Handling call
0807 11:10:52.266701 (+1698684us) spinlock_profiling.cc:243] Waited 4.08 ms on lock 0x7b5800000f98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc46061b0 00007f3fc4605a94 00007f3fbdcedc04 00007f3fbdcedb92
0807 11:10:52.341664 (+ 74963us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9390848}
I20260807 11:10:52.328375 29624 raft_consensus.cc:2781] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 11:10:52.328533 29510 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:52.350857 26046 raft_consensus.cc:2454] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:10:52.358161 29763 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:52.358760 26034 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:52.207047 29278 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:52.363619 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:52.370837 25932 raft_consensus.cc:3037] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:52.373031 26036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1685) took 3754 ms. Trace:
I20260807 11:10:52.347427 29787 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:52.340204 29634 raft_consensus.cc:2781] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:52.375451 25752 leader_election.cc:398] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Vote denied by peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009) with higher term. Message: Invalid argument: T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4.
I20260807 11:10:52.403450 25799 raft_consensus.cc:1226] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 4. Current term is 5. Ops: []
W20260807 11:10:52.336975 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 45068.9us
I20260807 11:10:52.375614 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:52.336632 25921 raft_consensus.cc:1226] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:52.407902 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1733) took 1559 ms. Trace:
I20260807 11:10:52.328141 26041 raft_consensus.cc:2396] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260807 11:10:52.409350 29779 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:52.385258 25981 leader_election.cc:401] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20260807 11:10:52.410773 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 84643.1us
I20260807 11:10:52.408268 25921 rpcz_store.cc:276] 0807 11:10:50.848381 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.336093 (+1487712us) service_pool.cc:227] Handling call
0807 11:10:52.336448 (+ 355us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:52.407602 (+ 71154us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:52.407841 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:52.411559 29670 raft_consensus.cc:2726] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465) with higher term. Message: Invalid argument: T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3.
I20260807 11:10:52.393389 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:52.402024 26038 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:10:52.403803 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:52.403245 26938 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:52.405349 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:52.407111 25869 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:52.414324 25798 raft_consensus.cc:2396] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260807 11:10:52.415179 25869 leader_election.cc:302] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:52.409575 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:52.377526 25806 raft_consensus.cc:1261] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260807 11:10:52.416580 26041 raft_consensus.cc:2396] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 11:10:52.411875 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:52.395051 29745 raft_consensus.cc:3037] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:52.430160 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:52.398267 26036 rpcz_store.cc:276] 0807 11:10:48.618626 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:48.625283 (+ 6657us) service_pool.cc:227] Handling call
0807 11:10:48.644121 (+ 18838us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:48.668980 (+ 24859us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:48.669020 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:48.686793 (+ 17773us) log.cc:842] Serialized 26 byte log entry
0807 11:10:48.829079 (+142286us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:10:48.855999 (+ 26920us) log.cc:842] Serialized 14 byte log entry
0807 11:10:48.856737 (+ 738us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:48.856954 (+ 217us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:48.937799 (+ 80845us) spinlock_profiling.cc:243] Waited 7.9 ms on lock 0x7b1800406f60. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbc79c638 00007f3fbc79fb11 00007f3fbc79fac9 00007f3fbc79fa59 00007f3fbc79fa21 00007f3fbc79ed1d 00007f3fc5a00204 00007f3fc5a00039 00007f3fbc77edc6
0807 11:10:52.292482 (+3354683us) spinlock_profiling.cc:243] Waited 7.79 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:10:52.372840 (+ 80358us) raft_consensus.cc:1671] finished
0807 11:10:52.372861 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:52.372972 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":29275,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":3396,"raft.queue_time_us":13803,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":3443,"spinlock_wait_cycles":36073472,"thread_start_us":95936,"threads_started":1,"wal-append.queue_time_us":783809}
I20260807 11:10:52.431775 26038 raft_consensus.cc:1226] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260807 11:10:52.437007 26034 raft_consensus.cc:3037] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:52.435596 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:52.437835 25806 raft_consensus.cc:2396] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260807 11:10:52.394874 29624 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:52.439414 29624 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:10:52.441004 25872 leader_election.cc:302] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:52.445600 25932 raft_consensus.cc:1261] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260807 11:10:52.447868 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:52.443897 25750 leader_election.cc:302] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:52.452033 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:52.455757 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:52.456698 25932 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:10:52.411710 29779 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:52.413938 29634 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:52.395588 29692 consensus_queue.cc:1046] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
I20260807 11:10:52.396617 29510 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:52.396546 29705 consensus_queue.cc:1035] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.065s
I20260807 11:10:52.492043 29029 consensus_queue.cc:1035] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20260807 11:10:52.547521 29029 consensus_queue.cc:1035] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:10:52.486382 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:52.549652 25796 raft_consensus.cc:2454] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 7.
I20260807 11:10:52.550873 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:52.551560 25796 raft_consensus.cc:2454] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:52.552335 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:52.552466 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:52.447454 29727 raft_consensus.cc:2781] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:52.553687 25919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1718) took 1878 ms. Trace:
I20260807 11:10:52.554122 25919 rpcz_store.cc:276] 0807 11:10:50.675414 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:50.783970 (+108556us) service_pool.cc:227] Handling call
0807 11:10:50.793274 (+ 9304us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:52.081610 (+1288336us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:52.553417 (+471807us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b54003395d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:52.553602 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2932798208}
I20260807 11:10:52.554708 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:52.555712 26986 raft_consensus.cc:2396] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260807 11:10:52.419096 29798 consensus_queue.cc:1035] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.137s
I20260807 11:10:50.535600 25800 rpcz_store.cc:276] 0807 11:10:49.180948 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:49.181111 (+ 163us) service_pool.cc:227] Handling call
0807 11:10:49.192379 (+ 11268us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:49.192597 (+ 218us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 11:10:49.692954 (+500357us) log.cc:842] Serialized 14 byte log entry
0807 11:10:50.534706 (+841752us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:50.534775 (+ 69us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:10:50.534844 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:50.534933 (+ 89us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:50.535090 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":944286,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":113,"thread_start_us":1177517,"threads_started":2}
I20260807 11:10:52.517053 26046 raft_consensus.cc:1261] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 11:10:52.557812 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:52.559101 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:52.500047 25799 raft_consensus.cc:1261] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 11:10:52.559978 25800 raft_consensus.cc:2396] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 11:10:52.455044 29593 consensus_queue.cc:1046] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.087s
W20260807 11:10:52.543540 29803 log.cc:924] Time spent T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.146s user 0.001s sys 0.000s
I20260807 11:10:52.451202 29719 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:52.441115 29636 consensus_queue.cc:1035] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.104s
I20260807 11:10:52.545250 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:52.555490 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:52.542960 25931 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1720) took 1867 ms (client timeout 1605 ms). Trace:
I20260807 11:10:52.556659 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:52.457648 25869 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:52.492297 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:52.456004 25872 leader_election.cc:302] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:52.589356 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:52.516371 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:52.442591 29695 consensus_queue.cc:1035] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.120s
I20260807 11:10:52.562557 25796 raft_consensus.cc:2379] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 5: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:52.563830 29510 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:52.515869 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:52.492250 29621 consensus_queue.cc:1035] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.080s
W20260807 11:10:52.501078 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 125754us
W20260807 11:10:52.574213 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 130409us
I20260807 11:10:52.500844 26980 raft_consensus.cc:1261] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 11:10:52.500808 26039 pending_rounds.cc:76] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9: Aborting all ops after (but not including) 2
I20260807 11:10:52.492826 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:52.578747 26034 raft_consensus.cc:2454] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:10:52.440810 29720 raft_consensus.cc:482] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:52.419374 29608 raft_consensus.cc:2726] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 11:10:52.459348 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:52.466759 25932 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 8.
I20260807 11:10:52.611364 25798 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:52.790686 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:52.791710 25931 rpcz_store.cc:269] 0807 11:10:50.676015 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:50.700398 (+ 24383us) service_pool.cc:227] Handling call
0807 11:10:52.542748 (+1842350us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b54003ff7d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:52.542883 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3290493184}
I20260807 11:10:52.793394 25752 leader_election.cc:302] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:52.796118 25797 raft_consensus.cc:3037] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:52.615682 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:52.797658 26036 raft_consensus.cc:1226] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260807 11:10:52.799924 26039 pending_rounds.cc:97] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260807 11:10:52.800362 26039 raft_consensus.cc:2866] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:10:52.575623 29634 raft_consensus.cc:3037] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:52.791170 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:52.819159 25799 raft_consensus.cc:2396] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260807 11:10:52.500439 29779 raft_consensus.cc:2781] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:10:52.876847 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:52.895052 26034 raft_consensus.cc:1226] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260807 11:10:52.443079 29278 raft_consensus.cc:2781] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:52.461275 29670 raft_consensus.cc:482] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:52.914266 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:52.759676 29623 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 3.550s user 0.042s sys 0.062s
I20260807 11:10:52.542979 29787 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:52.915767 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:53.801635 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1735) took 2952 ms. Trace:
W20260807 11:10:53.754300 26039 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.727s user 0.010s sys 0.009s
W20260807 11:10:53.713769 26938 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.653s user 0.001s sys 0.021s
W20260807 11:10:53.702630 26033 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59312 (request call id 1708) took 1637 ms (client timeout 2068 ms). Trace:
I20260807 11:10:53.802297 25917 rpcz_store.cc:276] 0807 11:10:50.848994 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.345573 (+1496579us) service_pool.cc:227] Handling call
0807 11:10:52.345850 (+ 277us) raft_consensus.cc:1457] Updating replica for 3 ops
0807 11:10:52.376846 (+ 30996us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:52.376884 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0807 11:10:52.381684 (+ 4800us) log.cc:842] Serialized 78 byte log entry
0807 11:10:52.584894 (+203210us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:10:52.616593 (+ 31699us) log.cc:842] Serialized 14 byte log entry
0807 11:10:52.621504 (+ 4911us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:52.621917 (+ 413us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:53.801234 (+1179317us) raft_consensus.cc:1671] finished
0807 11:10:53.801270 (+ 36us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:53.801541 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1740,"prepare.queue_time_us":31171,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":752,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":15875,"thread_start_us":143258,"threads_started":1,"wal-append.queue_time_us":374809}
I20260807 11:10:53.805600 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:53.805763 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:53.006848 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:10:53.724295 25918 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1716) took 3085 ms (client timeout 1681 ms). Trace:
I20260807 11:10:53.223515 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
W20260807 11:10:53.818495 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 1.3625e+06us
I20260807 11:10:53.489326 25808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 336) took 3020 ms. Trace:
W20260807 11:10:53.350368 26046 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.791s user 0.018s sys 0.040s
W20260807 11:10:53.803232 26039 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.776s user 0.011s sys 0.010s
I20260807 11:10:53.091578 25926 raft_consensus.cc:2454] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:10:53.808403 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:53.466281 25920 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1711) took 2838 ms (client timeout 1759 ms). Trace:
I20260807 11:10:53.107147 26036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1819) took 1022 ms. Trace:
W20260807 11:10:53.713862 26936 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.721s user 0.000s sys 0.014s
W20260807 11:10:53.821942 26936 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.834s user 0.001s sys 0.017s
I20260807 11:10:53.802901 25806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1720) took 1359 ms. Trace:
I20260807 11:10:53.104597 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 429) took 2594 ms. Trace:
I20260807 11:10:53.492394 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:53.102128 26984 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 458) took 2498 ms. Trace:
W20260807 11:10:52.932723 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 95497us
I20260807 11:10:53.812448 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1817) took 1739 ms. Trace:
I20260807 11:10:53.824664 26036 rpcz_store.cc:276] 0807 11:10:52.084859 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.437339 (+352480us) service_pool.cc:227] Handling call
0807 11:10:52.797365 (+360026us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:52.798142 (+ 777us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:53.106924 (+308782us) spinlock_profiling.cc:243] Waited 360 ms on lock 0x7b54000d0550. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:53.107077 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":826866432}
W20260807 11:10:53.804919 26033 rpcz_store.cc:269] 0807 11:10:52.065300 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.209414 (+144114us) service_pool.cc:227] Handling call
0807 11:10:53.702294 (+1492880us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x7b54000a3a54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:53.702534 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":37171456}
W20260807 11:10:53.821341 25920 rpcz_store.cc:269] 0807 11:10:50.627495 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:50.642800 (+ 15305us) service_pool.cc:227] Handling call
0807 11:10:53.465972 (+2823172us) spinlock_profiling.cc:243] Waited 85.3 ms on lock 0x7b54003b29d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:53.466186 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":196208512}
I20260807 11:10:53.826713 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:52.623823 29719 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:53.812865 29635 raft_consensus.cc:482] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:53.813062 26979 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 2.
I20260807 11:10:53.590683 26035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59312 (request call id 1700) took 2435 ms. Trace:
I20260807 11:10:53.843886 26035 rpcz_store.cc:276] 0807 11:10:51.155703 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:51.156801 (+ 1098us) service_pool.cc:227] Handling call
0807 11:10:53.590377 (+2433576us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b540022a954. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:53.590583 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2677189376}
I20260807 11:10:53.160631 26034 raft_consensus.cc:1226] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:52.615252 29267 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:53.106642 29624 consensus_meta.cc:287] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.667s user 0.006s sys 0.002s
I20260807 11:10:52.617071 29778 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:53.098959 29702 consensus_meta.cc:287] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.793s user 0.009s sys 0.001s
I20260807 11:10:53.177196 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:53.869545 25796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1547) took 1190 ms. Trace:
W20260807 11:10:53.818759 29623 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 4.628s user 0.055s sys 0.077s
I20260807 11:10:53.869920 25796 rpcz_store.cc:276] 0807 11:10:52.678770 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.115168 (+436398us) service_pool.cc:227] Handling call
0807 11:10:53.869479 (+754311us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:53.871094 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:53.871899 25796 raft_consensus.cc:2379] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 5: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:52.923012 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:53.846968 29779 raft_consensus.cc:686] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:53.872850 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:53.873250 26986 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 433) took 1293 ms. Trace:
I20260807 11:10:53.873795 25796 raft_consensus.cc:2396] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260807 11:10:53.874104 25932 raft_consensus.cc:2396] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 11:10:53.386274 29657 raft_consensus.cc:3032] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:53.874238 25796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1544) took 1290 ms. Trace:
I20260807 11:10:53.874529 29657 raft_consensus.cc:729] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:53.874699 25796 rpcz_store.cc:276] 0807 11:10:52.583537 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.872560 (+1289023us) service_pool.cc:227] Handling call
0807 11:10:53.874188 (+ 1628us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:53.875099 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1593) took 1289 ms. Trace:
I20260807 11:10:53.060293 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:53.614084 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:52.617435 29728 raft_consensus.cc:2726] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260807 11:10:53.876662 26975 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 426) took 1434 ms (client timeout 1857 ms). Trace:
W20260807 11:10:53.877060 26975 rpcz_store.cc:269] 0807 11:10:52.442318 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.454127 (+ 11809us) service_pool.cc:227] Handling call
0807 11:10:53.613834 (+1159707us) spinlock_profiling.cc:243] Waited 7.6 ms on lock 0x7b5800000f98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc46061b0 00007f3fc4605a94 00007f3fbdcedc04 00007f3fbdcedb92
0807 11:10:53.876590 (+262756us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17485056}
W20260807 11:10:53.877518 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 57304.1us
I20260807 11:10:53.878285 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:53.879094 26975 raft_consensus.cc:2379] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 6: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:53.879462 26975 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50960 (request call id 369) took 1180 ms. Trace:
I20260807 11:10:53.879776 26975 rpcz_store.cc:276] 0807 11:10:52.699462 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.878018 (+1178556us) service_pool.cc:227] Handling call
0807 11:10:53.879410 (+ 1392us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:52.741591 29516 raft_consensus.cc:2726] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 11:10:53.060644 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:10:53.819222 25918 rpcz_store.cc:269] 0807 11:10:50.639337 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:50.646196 (+ 6859us) service_pool.cc:227] Handling call
0807 11:10:53.724016 (+3077820us) spinlock_profiling.cc:243] Waited 32.7 ms on lock 0x7b5400053054. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:53.724186 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":75295616}
W20260807 11:10:53.778497 26937 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.813s user 0.005s sys 0.019s
W20260807 11:10:53.820647 26046 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.261s user 0.029s sys 0.045s
W20260807 11:10:53.821183 26980 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50960 (request call id 366) took 1234 ms (client timeout 1552 ms). Trace:
I20260807 11:10:53.103102 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1816) took 1030 ms. Trace:
I20260807 11:10:53.822361 25806 rpcz_store.cc:276] 0807 11:10:52.443209 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.443325 (+ 116us) service_pool.cc:227] Handling call
0807 11:10:52.443667 (+ 342us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:52.490040 (+ 46373us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:52.490090 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:52.490794 (+ 704us) log.cc:842] Serialized 26 byte log entry
0807 11:10:52.682208 (+191414us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:10:52.682263 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:52.682374 (+ 111us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:53.415337 (+732963us) raft_consensus.cc:1671] finished
0807 11:10:53.415366 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:53.802840 (+387474us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30778,"raft.queue_time_us":49679,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":34348,"thread_start_us":106823,"threads_started":1,"wal-append.queue_time_us":860512}
W20260807 11:10:53.218673 25981 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.120.132:43279 timed out after 1.827s (SENT) [suppressed 1 similar messages]
I20260807 11:10:53.820444 25808 rpcz_store.cc:276] 0807 11:10:50.469307 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:50.469466 (+ 159us) service_pool.cc:227] Handling call
0807 11:10:50.502831 (+ 33365us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:50.514461 (+ 11630us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:50.514490 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:50.514526 (+ 36us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:50.514569 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:53.489025 (+2974456us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b54003acfd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:53.489084 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:53.489237 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26268032}
I20260807 11:10:53.825019 26984 rpcz_store.cc:276] 0807 11:10:50.603823 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:50.630975 (+ 27152us) service_pool.cc:227] Handling call
0807 11:10:50.646385 (+ 15410us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:50.844386 (+198001us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:53.101881 (+2257495us) spinlock_profiling.cc:243] Waited 191 ms on lock 0x7b54006a04d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:53.102065 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":440410496}
I20260807 11:10:53.495728 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1696) took 3002 ms. Trace:
I20260807 11:10:53.821475 25798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1736) took 1198 ms. Trace:
I20260807 11:10:53.842054 25803 raft_consensus.cc:2396] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 11:10:53.825361 26041 rpcz_store.cc:276] 0807 11:10:52.073235 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.431244 (+358009us) service_pool.cc:227] Handling call
0807 11:10:52.486715 (+ 55471us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:52.489724 (+ 3009us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:52.489787 (+ 63us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:52.615197 (+125410us) log.cc:842] Serialized 26 byte log entry
0807 11:10:52.875602 (+260405us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:52.875649 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:52.875732 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:53.457047 (+581315us) raft_consensus.cc:1671] finished
0807 11:10:53.457072 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:53.810411 (+353339us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2462,"raft.queue_time_us":3030,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":152,"thread_start_us":168681,"threads_started":1,"wal-append.queue_time_us":735234}
I20260807 11:10:53.845840 25920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1599) took 1143 ms. Trace:
I20260807 11:10:53.846597 26033 raft_consensus.cc:1226] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260807 11:10:52.621460 29727 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:53.842279 25800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1545) took 1258 ms. Trace:
I20260807 11:10:53.914454 25920 rpcz_store.cc:276] 0807 11:10:52.702516 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.826439 (+1123923us) service_pool.cc:227] Handling call
0807 11:10:53.845776 (+ 19337us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:53.494973 29802 raft_consensus.cc:3032] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:53.920044 25917 raft_consensus.cc:2454] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 5.
I20260807 11:10:52.875288 29798 raft_consensus.cc:2781] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election won for term 6
W20260807 11:10:53.611020 29807 log.cc:924] Time spent T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 1.091s user 0.000s sys 0.014s
W20260807 11:10:53.099628 29745 consensus_meta.cc:287] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.608s user 0.008s sys 0.001s
W20260807 11:10:53.498739 29718 log.cc:924] Time spent T 00000000000000000000000000000000 P 98664bdefe2a4a829847688b8d0d50c1: Append to log took a long time: real 2.848s user 0.015s sys 0.025s
W20260807 11:10:53.746730 29736 log.cc:924] Time spent T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 2.910s user 0.012s sys 0.004s
I20260807 11:10:53.854270 25916 raft_consensus.cc:1226] T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260807 11:10:53.854379 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:53.861492 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1826) took 1710 ms. Trace:
I20260807 11:10:53.930723 26034 rpcz_store.cc:276] 0807 11:10:52.151074 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.030584 (+879510us) service_pool.cc:227] Handling call
0807 11:10:53.160270 (+129686us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:53.861155 (+700885us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:53.861417 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:53.863884 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:53.931418 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:53.823585 26037 rpcz_store.cc:276] 0807 11:10:50.510762 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:50.510943 (+ 181us) service_pool.cc:227] Handling call
0807 11:10:50.514136 (+ 3193us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:50.514432 (+ 296us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 11:10:50.549639 (+ 35207us) log.cc:842] Serialized 14 byte log entry
0807 11:10:53.104196 (+2554557us) spinlock_profiling.cc:243] Waited 5.8 ms on lock 0x7b18003a44a0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbf35382c 00007f3fbc77e9dc 00007f3fc55b5a39 00007f3fbc79b8bb 00007f3fbc799cbf 00007f3fbc799155 00007f3fbf6bc3d0 00007f3fbf6ce576 00007f3fbf6df0d7
0807 11:10:53.104289 (+ 93us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:53.104336 (+ 47us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:10:53.104386 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:53.104451 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:53.104546 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6171,"prepare.queue_time_us":125382,"prepare.run_cpu_time_us":11195,"prepare.run_wall_time_us":17072,"raft.queue_time_us":134459,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84,"spinlock_wait_cycles":13349760,"thread_start_us":141841,"threads_started":3,"wal-append.queue_time_us":96310,"wal-append.run_cpu_time_us":11322,"wal-append.run_wall_time_us":1179528}
I20260807 11:10:53.864616 29635 raft_consensus.cc:504] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:53.866525 29719 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:52.875720 29720 raft_consensus.cc:504] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:53.848742 29636 raft_consensus.cc:2781] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:53.867626 25786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 341) took 1499 ms. Trace:
I20260807 11:10:53.861979 29217 consensus_queue.cc:260] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:53.933362 29636 raft_consensus.cc:482] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
W20260807 11:10:53.803843 26938 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.743s user 0.001s sys 0.024s
I20260807 11:10:53.093540 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:53.873625 29779 consensus_queue.cc:237] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:53.943617 26987 raft_consensus.cc:2354] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6.
I20260807 11:10:53.931841 26035 raft_consensus.cc:2454] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:53.875779 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:53.095882 29205 consensus_meta.cc:287] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.777s user 0.007s sys 0.005s
W20260807 11:10:53.061286 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 603688us
I20260807 11:10:53.944700 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:10:53.742806 29691 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.202s user 0.000s sys 0.010s
I20260807 11:10:53.815130 25931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35276 (request call id 290) took 1432 ms. Trace:
W20260807 11:10:53.479375 29821 log.cc:924] Time spent T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.960s user 0.003s sys 0.001s
I20260807 11:10:53.915084 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1830) took 1737 ms. Trace:
I20260807 11:10:53.945506 25931 rpcz_store.cc:276] 0807 11:10:52.382329 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.792634 (+410305us) service_pool.cc:227] Handling call
0807 11:10:53.815074 (+1022440us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:53.914440 25798 rpcz_store.cc:276] 0807 11:10:52.622910 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.589368 (+966458us) service_pool.cc:227] Handling call
0807 11:10:53.821416 (+232048us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:53.946084 25869 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:53.946607 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:53.945793 26033 rpcz_store.cc:276] 0807 11:10:52.177299 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.846046 (+1668747us) service_pool.cc:227] Handling call
0807 11:10:53.846389 (+ 343us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:53.914815 (+ 68426us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:53.915030 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:53.947237 25931 raft_consensus.cc:2454] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:10:53.946187 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:53.947710 25931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35276 (request call id 283) took 1571 ms. Trace:
W20260807 11:10:53.916386 26937 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.951s user 0.006s sys 0.020s
I20260807 11:10:53.915583 26042 rpcz_store.cc:276] 0807 11:10:50.494026 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:50.502790 (+ 8764us) service_pool.cc:227] Handling call
0807 11:10:50.517385 (+ 14595us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:50.828439 (+311054us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:53.495425 (+2666986us) spinlock_profiling.cc:243] Waited 279 ms on lock 0x7b5400313654. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:53.495655 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":641938176}
I20260807 11:10:53.948102 25931 rpcz_store.cc:276] 0807 11:10:52.376233 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.946374 (+1570141us) service_pool.cc:227] Handling call
0807 11:10:53.947660 (+ 1286us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:10:53.917624 26980 rpcz_store.cc:269] 0807 11:10:52.586945 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.799810 (+212865us) service_pool.cc:227] Handling call
0807 11:10:53.821094 (+1021284us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:53.915692 26038 rpcz_store.cc:276] 0807 11:10:52.072563 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.417649 (+345086us) service_pool.cc:227] Handling call
0807 11:10:52.418522 (+ 873us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:52.437205 (+ 18683us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:53.102875 (+665670us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b54000a4e54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:53.103046 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":29906816}
I20260807 11:10:53.915118 25800 rpcz_store.cc:276] 0807 11:10:52.583943 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.492057 (+908114us) service_pool.cc:227] Handling call
0807 11:10:53.842191 (+350134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:53.921119 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:53.921573 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1604) took 1217 ms. Trace:
I20260807 11:10:53.949493 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:53.949841 25917 rpcz_store.cc:276] 0807 11:10:52.704129 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.805465 (+1101336us) service_pool.cc:227] Handling call
0807 11:10:53.921517 (+116052us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:53.455224 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:53.947724 25869 leader_election.cc:302] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:53.952152 25918 raft_consensus.cc:2379] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 6: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:53.875536 25932 rpcz_store.cc:276] 0807 11:10:52.585401 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.915437 (+330036us) service_pool.cc:227] Handling call
0807 11:10:53.874917 (+959480us) spinlock_profiling.cc:243] Waited 71.4 ms on lock 0x7b54001a1b54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:53.875047 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":164331904}
W20260807 11:10:53.853140 29682 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.307s user 0.006s sys 0.013s
I20260807 11:10:53.922574 25803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1546) took 1244 ms. Trace:
I20260807 11:10:53.923317 26985 raft_consensus.cc:2396] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20260807 11:10:53.923408 26979 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 425) took 1501 ms (client timeout 1878 ms). Trace:
I20260807 11:10:53.926659 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:53.821034 29761 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:53.928682 26936 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:53.845189 29278 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:53.948135 26035 raft_consensus.cc:2454] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
W20260807 11:10:53.336944 29639 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 4.126s user 0.031s sys 0.062s
I20260807 11:10:53.952229 25920 raft_consensus.cc:2396] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260807 11:10:53.952713 25918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1594) took 1356 ms. Trace:
I20260807 11:10:53.964221 25918 rpcz_store.cc:276] 0807 11:10:52.596656 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.916967 (+1320311us) service_pool.cc:227] Handling call
0807 11:10:53.952659 (+ 35692us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:53.964831 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:53.951069 29691 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.411s user 0.001s sys 0.012s
I20260807 11:10:53.973837 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:53.875519 29657 consensus_queue.cc:260] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:53.880671 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:53.953279 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:53.978807 25932 raft_consensus.cc:2379] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 5: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:53.979152 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1600) took 1276 ms. Trace:
I20260807 11:10:53.883848 29624 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:53.951625 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:53.985224 26984 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 432) took 1410 ms. Trace:
I20260807 11:10:53.964324 29278 raft_consensus.cc:3037] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:53.933400 25786 rpcz_store.cc:276] 0807 11:10:52.368368 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.370056 (+ 1688us) service_pool.cc:227] Handling call
0807 11:10:52.374126 (+ 4070us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:52.374506 (+ 380us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:52.374538 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:52.375045 (+ 507us) log.cc:842] Serialized 26 byte log entry
0807 11:10:52.721859 (+346814us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:10:52.721917 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:52.722016 (+ 99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:53.867418 (+1145402us) raft_consensus.cc:1671] finished
0807 11:10:53.867442 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:53.867568 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4837,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":8908,"spinlock_wait_cycles":25344,"thread_start_us":346323,"threads_started":1,"wal-append.queue_time_us":1487035}
I20260807 11:10:53.921351 29727 raft_consensus.cc:3037] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:53.953853 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:53.951361 25799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1728) took 1375 ms. Trace:
I20260807 11:10:53.955507 25869 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:53.933931 29636 raft_consensus.cc:3037] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:53.955054 25796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1739) took 1189 ms. Trace:
I20260807 11:10:53.990724 25799 rpcz_store.cc:276] 0807 11:10:52.575679 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.591136 (+1015457us) service_pool.cc:227] Handling call
0807 11:10:53.951292 (+360156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:53.956314 29844 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:53.955971 25803 rpcz_store.cc:276] 0807 11:10:52.678422 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.150643 (+472221us) service_pool.cc:227] Handling call
0807 11:10:53.922490 (+771847us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:10:53.927196 29558 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.813s user 0.013s sys 0.024s
I20260807 11:10:53.925242 29802 raft_consensus.cc:729] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:53.960965 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:53.959548 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:53.997844 26035 raft_consensus.cc:3037] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:53.994290 29730 raft_consensus.cc:2781] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 11:10:53.998013 29609 consensus_queue.cc:1035] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:10:53.998569 29778 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:53.998658 29730 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:53.999204 29730 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:10:53.961544 26985 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50960 (request call id 365) took 1375 ms. Trace:
I20260807 11:10:53.999392 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:53.957502 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:53.957857 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:54.000643 25808 raft_consensus.cc:2379] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 5: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
W20260807 11:10:53.956154 26979 rpcz_store.cc:269] 0807 11:10:52.422231 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.424410 (+ 2179us) service_pool.cc:227] Handling call
0807 11:10:53.923243 (+1498833us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540068c254. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:53.923349 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2345805696}
I20260807 11:10:53.959692 25806 raft_consensus.cc:2379] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 5: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:53.965853 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:54.006455 26980 raft_consensus.cc:2379] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 3: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:54.006523 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:53.933809 29217 raft_consensus.cc:3037] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
W20260807 11:10:53.966289 25797 consensus_meta.cc:287] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.950s user 0.008s sys 0.000s
I20260807 11:10:54.007748 25797 raft_consensus.cc:2454] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:10:54.008184 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:54.008283 25797 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1718) took 1575 ms (client timeout 1887 ms). Trace:
I20260807 11:10:53.841773 29670 raft_consensus.cc:504] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:54.015671 29635 leader_election.cc:288] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:54.016938 29670 leader_election.cc:288] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:53.952425 26938 leader_election.cc:302] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:54.018251 25872 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:10:53.950896 26978 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 457) took 3389 ms (client timeout 1958 ms). Trace:
I20260807 11:10:53.966997 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:53.929888 29323 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:53.957444 29719 consensus_queue.cc:1046] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260807 11:10:53.960470 29728 raft_consensus.cc:2726] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:53.969861 29417 raft_consensus.cc:2726] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260807 11:10:53.971506 25926 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1712) took 3336 ms (client timeout 2191 ms). Trace:
I20260807 11:10:53.972548 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:10:54.028970 25926 rpcz_store.cc:269] 0807 11:10:50.635704 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:50.635899 (+ 195us) service_pool.cc:227] Handling call
0807 11:10:53.971333 (+3335434us) spinlock_profiling.cc:243] Waited 2.05 s on lock 0x7b54003b1ad4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:53.971436 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4718718848}
I20260807 11:10:53.873627 26986 rpcz_store.cc:276] 0807 11:10:52.580009 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.819236 (+239227us) service_pool.cc:227] Handling call
0807 11:10:53.873190 (+1053954us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:53.934183 29720 leader_election.cc:288] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:53.974906 25920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1589) took 1416 ms. Trace:
I20260807 11:10:54.033812 25920 rpcz_store.cc:276] 0807 11:10:52.558432 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.915497 (+1357065us) service_pool.cc:227] Handling call
0807 11:10:53.974836 (+ 59339us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:53.951337 29173 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:53.979390 25932 rpcz_store.cc:276] 0807 11:10:52.702944 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.953011 (+1250067us) service_pool.cc:227] Handling call
0807 11:10:53.979111 (+ 26100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.037374 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:53.950346 26038 raft_consensus.cc:2379] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 5: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
W20260807 11:10:53.963973 29639 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 4.779s user 0.035s sys 0.073s
I20260807 11:10:53.954852 29813 raft_consensus.cc:2726] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 11:10:53.996515 29844 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:53.955895 29682 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.410s user 0.007s sys 0.014s
I20260807 11:10:53.960001 29830 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:53.985558 26984 rpcz_store.cc:276] 0807 11:10:52.575211 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.925354 (+1350143us) service_pool.cc:227] Handling call
0807 11:10:53.985164 (+ 59810us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.042819 29830 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:54.043870 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:54.045212 29830 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:54.045418 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:53.947813 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:54.047714 26033 raft_consensus.cc:2396] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260807 11:10:54.048686 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:54.049786 26033 raft_consensus.cc:2396] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 11:10:54.051473 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:53.986487 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:53.991176 25796 rpcz_store.cc:276] 0807 11:10:52.765384 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.875493 (+1110109us) service_pool.cc:227] Handling call
0807 11:10:53.954988 (+ 79495us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:53.991782 25917 raft_consensus.cc:2454] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 6.
I20260807 11:10:54.053632 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1605) took 1349 ms. Trace:
I20260807 11:10:54.054061 25917 rpcz_store.cc:276] 0807 11:10:52.704401 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.953635 (+1249234us) service_pool.cc:227] Handling call
0807 11:10:54.053561 (+ 99926us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:53.943534 29745 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:53.983595 29657 raft_consensus.cc:3037] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
W20260807 11:10:54.035858 29843 log.cc:924] Time spent T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
I20260807 11:10:53.993455 25801 raft_consensus.cc:1226] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 7. Current term is 8. Ops: []
I20260807 11:10:53.993462 25869 leader_election.cc:302] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:54.059723 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 66344.4us
W20260807 11:10:53.996738 26987 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50960 (request call id 361) took 1444 ms (client timeout 1812 ms). Trace:
I20260807 11:10:53.996912 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:54.007030 26980 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50960 (request call id 368) took 1419 ms. Trace:
I20260807 11:10:54.060909 25803 raft_consensus.cc:3037] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:54.060876 26980 rpcz_store.cc:276] 0807 11:10:52.587955 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.949061 (+1361106us) service_pool.cc:227] Handling call
0807 11:10:54.006974 (+ 57913us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.008749 25806 raft_consensus.cc:2379] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 5: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
W20260807 11:10:54.008692 25797 rpcz_store.cc:269] 0807 11:10:52.433332 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.433488 (+ 156us) service_pool.cc:227] Handling call
0807 11:10:54.008225 (+1574737us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.009035 25931 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 2.
I20260807 11:10:54.008582 25916 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1724) took 3322 ms. Trace:
W20260807 11:10:54.062461 25931 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35276 (request call id 284) took 1686 ms (client timeout 2244 ms). Trace:
I20260807 11:10:53.966277 29761 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:54.062978 25931 rpcz_store.cc:269] 0807 11:10:52.376511 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.959313 (+1582802us) service_pool.cc:227] Handling call
0807 11:10:54.062398 (+103085us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.062423 25916 rpcz_store.cc:276] 0807 11:10:50.686599 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.076073 (+1389474us) service_pool.cc:227] Handling call
0807 11:10:52.076787 (+ 714us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:53.930518 (+1853731us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.008330 (+ 77812us) spinlock_profiling.cc:243] Waited 1.77 s on lock 0x7b5400185954. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:54.008523 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4065771904}
W20260807 11:10:54.060655 26987 rpcz_store.cc:269] 0807 11:10:52.551863 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.552039 (+ 176us) service_pool.cc:227] Handling call
0807 11:10:53.995519 (+1443480us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54006b03d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:53.996678 (+ 1159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3168703232}
I20260807 11:10:54.063956 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:54.014839 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:54.064699 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:53.961308 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:54.065133 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:54.018747 25799 raft_consensus.cc:2396] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 11:10:54.020381 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1734) took 3171 ms. Trace:
I20260807 11:10:54.065913 25925 rpcz_store.cc:276] 0807 11:10:50.848691 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.343960 (+1495269us) service_pool.cc:227] Handling call
0807 11:10:52.364432 (+ 20472us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:52.364822 (+ 390us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:52.364850 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:52.365310 (+ 460us) log.cc:842] Serialized 26 byte log entry
0807 11:10:52.440991 (+ 75681us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:52.441034 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:52.441109 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:53.478676 (+1037567us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:10:53.992809 (+514133us) raft_consensus.cc:1671] finished
0807 11:10:53.992832 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:54.020313 (+ 27481us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3378,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":92,"spinlock_wait_cycles":31729408,"thread_start_us":73308,"threads_started":1,"wal-append.queue_time_us":153261}
W20260807 11:10:54.020660 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 68336.2us
I20260807 11:10:54.067310 26936 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:54.020665 29609 consensus_queue.cc:1035] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.021s
I20260807 11:10:54.069557 25801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 347) took 1694 ms. Trace:
I20260807 11:10:54.069950 25801 rpcz_store.cc:276] 0807 11:10:52.374887 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.377155 (+ 2268us) service_pool.cc:227] Handling call
0807 11:10:52.396352 (+ 19197us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:54.059274 (+1662922us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.069334 (+ 10060us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b540014ebd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:54.069500 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3672509312}
I20260807 11:10:54.024281 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:53.994499 29558 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.880s user 0.014s sys 0.024s
I20260807 11:10:53.973069 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:54.075951 26034 raft_consensus.cc:2454] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:54.027554 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:54.030318 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:54.016938 29635 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:54.031706 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:54.078356 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:54.005342 26975 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 459) took 1194 ms. Trace:
I20260807 11:10:54.032857 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:54.036031 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:54.078964 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:54.079574 25808 raft_consensus.cc:3037] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:54.079676 25920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1601) took 1376 ms. Trace:
I20260807 11:10:54.026113 29778 raft_consensus.cc:2781] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:10:54.081393 25918 raft_consensus.cc:2454] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:54.082332 25925 raft_consensus.cc:2379] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 6: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:54.082592 25918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1591) took 1506 ms. Trace:
I20260807 11:10:54.082799 25921 raft_consensus.cc:2379] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 6: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:54.082765 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:54.083017 25918 rpcz_store.cc:276] 0807 11:10:52.575974 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.965000 (+1389026us) service_pool.cc:227] Handling call
0807 11:10:54.082505 (+117505us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.084074 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:54.084088 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:54.057677 25800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1733) took 1435 ms. Trace:
I20260807 11:10:54.055811 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:54.085192 25918 raft_consensus.cc:3037] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:54.085227 25800 rpcz_store.cc:276] 0807 11:10:52.621964 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.977304 (+1355340us) service_pool.cc:227] Handling call
0807 11:10:54.057624 (+ 80320us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.060395 25869 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:54.086532 25869 leader_election.cc:302] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:54.064882 26979 raft_consensus.cc:2454] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:53.999785 26985 rpcz_store.cc:276] 0807 11:10:52.586591 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.875656 (+289065us) service_pool.cc:227] Handling call
0807 11:10:53.961489 (+1085833us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.068732 25799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1731) took 1447 ms. Trace:
I20260807 11:10:54.088158 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:54.069247 25931 raft_consensus.cc:2379] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 8: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:54.057165 29445 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:54.073173 26037 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:54.073951 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:54.073894 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:54.036375 29644 raft_consensus.cc:3032] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Stepping down as leader of term 4
W20260807 11:10:54.020046 26978 rpcz_store.cc:269] 0807 11:10:50.561821 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:50.577197 (+ 15376us) service_pool.cc:227] Handling call
0807 11:10:53.454927 (+2877730us) spinlock_profiling.cc:243] Waited 21.9 ms on lock 0x7b5800000f98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc46061b0 00007f3fc4605a94 00007f3fbdcedc04 00007f3fbdcedb92
0807 11:10:53.950802 (+495875us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":50342144}
W20260807 11:10:54.059617 29634 consensus_meta.cc:287] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 1.045s user 0.012s sys 0.000s
I20260807 11:10:54.080120 25920 rpcz_store.cc:276] 0807 11:10:52.703262 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.034729 (+1331467us) service_pool.cc:227] Handling call
0807 11:10:54.079619 (+ 44890us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.080257 25786 raft_consensus.cc:2379] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 8: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:54.077459 29838 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:54.081373 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:54.084118 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:54.046782 26035 raft_consensus.cc:2454] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:10:53.046690 29799 consensus_queue.cc:1035] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.898s
I20260807 11:10:54.087864 29635 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:54.089011 25931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1588) took 1530 ms. Trace:
I20260807 11:10:54.106897 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:54.107339 25931 rpcz_store.cc:276] 0807 11:10:52.558140 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.063735 (+1505595us) service_pool.cc:227] Handling call
0807 11:10:54.088957 (+ 25222us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.107690 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:54.108278 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:54.024353 26041 raft_consensus.cc:2396] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260807 11:10:54.087565 25799 rpcz_store.cc:276] 0807 11:10:52.621351 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.991417 (+1370066us) service_pool.cc:227] Handling call
0807 11:10:54.068651 (+ 77234us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.069027 26936 leader_election.cc:302] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:54.103644 25926 raft_consensus.cc:2454] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 6.
I20260807 11:10:54.076668 29267 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:54.089730 25869 leader_election.cc:302] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:54.091029 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1596) took 1412 ms. Trace:
I20260807 11:10:54.023701 29323 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:54.111686 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:54.112190 25869 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:54.092504 26038 raft_consensus.cc:2454] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:54.114384 26038 raft_consensus.cc:1203] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[3.2-6.4] Dedup: 3.2->[4.3-6.4]
I20260807 11:10:54.114786 26038 pending_rounds.cc:76] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9: Aborting all ops after (but not including) 2
I20260807 11:10:54.115028 26038 pending_rounds.cc:97] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260807 11:10:54.115242 26038 raft_consensus.cc:2866] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:10:54.039716 29173 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:54.097049 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:54.069742 29720 raft_consensus.cc:2726] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:54.098340 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:54.101918 29778 raft_consensus.cc:686] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:54.128983 26979 raft_consensus.cc:1226] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:54.045722 29844 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:54.100714 25808 raft_consensus.cc:2454] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 5.
I20260807 11:10:54.102396 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:54.071025 29830 raft_consensus.cc:2726] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 11:10:54.131098 26987 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50960 (request call id 364) took 1566 ms. Trace:
I20260807 11:10:54.131510 26987 rpcz_store.cc:276] 0807 11:10:52.564704 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.085100 (+1520396us) service_pool.cc:227] Handling call
0807 11:10:54.131038 (+ 45938us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.080181 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:54.104166 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:54.105347 25872 leader_election.cc:302] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:54.104830 26986 raft_consensus.cc:1226] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260807 11:10:54.110021 25926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1590) took 1551 ms. Trace:
I20260807 11:10:54.106863 25797 raft_consensus.cc:3037] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:54.133630 25926 rpcz_store.cc:276] 0807 11:10:52.558777 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.032603 (+1473826us) service_pool.cc:227] Handling call
0807 11:10:54.109956 (+ 77353us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.134760 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:54.110598 25925 rpcz_store.cc:276] 0807 11:10:52.678357 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.067362 (+1389005us) service_pool.cc:227] Handling call
0807 11:10:54.090984 (+ 23622us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.091799 29608 raft_consensus.cc:2781] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:54.138178 26034 raft_consensus.cc:2396] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260807 11:10:54.138582 29841 log.cc:924] Time spent T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
I20260807 11:10:54.138746 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:54.139283 26034 raft_consensus.cc:1226] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 5. Current term is 6. Ops: []
I20260807 11:10:54.139403 25796 raft_consensus.cc:2454] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 5.
I20260807 11:10:54.139868 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 433) took 1878 ms. Trace:
I20260807 11:10:54.140223 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:54.140177 26034 rpcz_store.cc:276] 0807 11:10:52.261525 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.138827 (+1877302us) service_pool.cc:227] Handling call
0807 11:10:54.139119 (+ 292us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:54.139664 (+ 545us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.139821 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.092406 26033 raft_consensus.cc:3037] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:54.149890 29840 raft_consensus.cc:2781] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:54.150364 29840 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:54.150692 29840 raft_consensus.cc:3037] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:54.150913 29323 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:54.082067 26975 rpcz_store.cc:276] 0807 11:10:52.802840 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.880402 (+1077562us) service_pool.cc:227] Handling call
0807 11:10:53.996950 (+116548us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.091552 29813 raft_consensus.cc:2781] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:54.153246 29813 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:54.153632 29813 raft_consensus.cc:3037] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:54.153733 29745 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:54.112322 29838 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:54.158730 26034 raft_consensus.cc:1261] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260807 11:10:54.118968 25921 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35276 (request call id 286) took 1741 ms (client timeout 1521 ms). Trace:
I20260807 11:10:54.163775 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 434) took 1901 ms. Trace:
I20260807 11:10:54.164134 29173 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:54.164211 26034 rpcz_store.cc:276] 0807 11:10:52.262059 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.158057 (+1895998us) service_pool.cc:227] Handling call
0807 11:10:54.158417 (+ 360us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:54.163456 (+ 5039us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.163710 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.128527 29624 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:54.165036 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:54.109581 29634 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:54.165926 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1595) took 1524 ms. Trace:
I20260807 11:10:54.166245 25925 rpcz_store.cc:276] 0807 11:10:52.641357 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.136056 (+1494699us) service_pool.cc:227] Handling call
0807 11:10:54.165867 (+ 29811us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.167310 29634 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:54.167371 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:54.113049 25931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1602) took 1405 ms. Trace:
I20260807 11:10:54.168195 25931 rpcz_store.cc:276] 0807 11:10:52.703546 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.108072 (+1404526us) service_pool.cc:227] Handling call
0807 11:10:54.108963 (+ 891us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.168699 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1822) took 2072 ms. Trace:
I20260807 11:10:54.169292 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:54.170377 29830 consensus_queue.cc:1046] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 11:10:54.169114 26039 rpcz_store.cc:276] 0807 11:10:52.096592 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.491478 (+394886us) service_pool.cc:227] Handling call
0807 11:10:52.500579 (+ 9101us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:52.905423 (+404844us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:52.905472 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:53.026891 (+121419us) log.cc:842] Serialized 26 byte log entry
0807 11:10:53.822213 (+795322us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:53.822257 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:53.822327 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:54.168500 (+346173us) raft_consensus.cc:1671] finished
0807 11:10:54.168520 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:54.168642 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":793455,"threads_started":1,"wal-append.queue_time_us":947288}
I20260807 11:10:54.171442 26042 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 7.
I20260807 11:10:54.171574 29830 consensus_queue.cc:1035] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 11:10:54.171931 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:54.115861 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:54.101007 29669 raft_consensus.cc:2726] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:54.101223 29644 raft_consensus.cc:729] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:54.173650 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 440) took 1909 ms. Trace:
I20260807 11:10:54.174149 26037 raft_consensus.cc:1261] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 11:10:54.168438 25925 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:54.174921 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 431) took 1914 ms. Trace:
W20260807 11:10:54.163565 25921 rpcz_store.cc:269] 0807 11:10:52.377165 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.411628 (+ 34463us) service_pool.cc:227] Handling call
0807 11:10:54.118712 (+1707084us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b540033aed4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:54.118881 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3829825408}
I20260807 11:10:54.057611 29802 consensus_queue.cc:260] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:54.175762 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:54.176345 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:54.176556 29802 raft_consensus.cc:3037] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:54.177129 25921 raft_consensus.cc:2379] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 6: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:54.081641 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:54.096887 29445 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:54.178305 25806 raft_consensus.cc:3037] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:54.179646 29445 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:54.160983 29840 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:54.177924 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:54.123677 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:54.170467 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:54.174083 26039 rpcz_store.cc:276] 0807 11:10:52.264447 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.172907 (+1908460us) service_pool.cc:227] Handling call
0807 11:10:54.173189 (+ 282us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:54.173363 (+ 174us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:54.173388 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:54.173425 (+ 37us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:10:54.173463 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.173512 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:54.173596 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.126684 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:54.137650 29846 log.cc:924] Time spent T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.117s user 0.000s sys 0.001s
I20260807 11:10:54.144090 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:54.208007 25796 raft_consensus.cc:2454] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:54.208636 25796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1738) took 1585 ms. Trace:
I20260807 11:10:54.208990 25796 rpcz_store.cc:276] 0807 11:10:52.623533 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.143834 (+1520301us) service_pool.cc:227] Handling call
0807 11:10:54.208539 (+ 64705us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.210007 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:54.210867 25796 raft_consensus.cc:2396] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260807 11:10:54.211261 25796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1735) took 1588 ms. Trace:
I20260807 11:10:54.211557 25796 rpcz_store.cc:276] 0807 11:10:52.622622 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.209660 (+1587038us) service_pool.cc:227] Handling call
0807 11:10:54.211209 (+ 1549us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.146158 29635 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:54.159559 29838 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:54.112836 29609 raft_consensus.cc:2726] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260807 11:10:54.161078 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 57463.4us
I20260807 11:10:54.161777 29720 raft_consensus.cc:2781] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:54.161434 29778 consensus_queue.cc:237] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:54.153759 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 85332us
I20260807 11:10:54.162665 25916 raft_consensus.cc:3037] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:54.172919 29267 raft_consensus.cc:3032] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 11:10:54.173154 25786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1727) took 1612 ms. Trace:
I20260807 11:10:54.269213 25786 rpcz_store.cc:276] 0807 11:10:52.561050 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.105288 (+1544238us) service_pool.cc:227] Handling call
0807 11:10:54.173097 (+ 67809us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.270675 26936 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:54.173825 25917 raft_consensus.cc:2379] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 6: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:54.174271 29644 consensus_queue.cc:260] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:54.124361 25801 raft_consensus.cc:2454] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:54.273325 25806 raft_consensus.cc:2454] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:54.282239 25801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1548) took 1574 ms. Trace:
I20260807 11:10:54.282578 25801 rpcz_store.cc:276] 0807 11:10:52.707935 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.070823 (+1362888us) service_pool.cc:227] Handling call
0807 11:10:54.282181 (+211358us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.116132 29417 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:54.122908 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:54.125849 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:54.284384 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 1.06566e+06us
I20260807 11:10:54.284737 25800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1730) took 1691 ms. Trace:
I20260807 11:10:54.181851 26938 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:54.182756 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 436) took 1919 ms. Trace:
I20260807 11:10:54.184480 25921 raft_consensus.cc:2396] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260807 11:10:54.286063 25797 raft_consensus.cc:2454] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:10:54.137380 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:54.287925 25808 raft_consensus.cc:3032] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 11:10:54.288347 25808 raft_consensus.cc:729] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
W20260807 11:10:54.287787 25981 connection.cc:232] RPC call timeout handler was delayed by -6.7109419209882617s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.196s for a response
I20260807 11:10:54.116789 25918 raft_consensus.cc:2454] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 6.
W20260807 11:10:54.289145 25981 connection.cc:232] RPC call timeout handler was delayed by -6.64164192089811s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2037s for a response
W20260807 11:10:54.289546 25981 connection.cc:232] RPC call timeout handler was delayed by -6.6172727658413351s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1809s for a response
W20260807 11:10:54.289814 25981 connection.cc:232] RPC call timeout handler was delayed by -6.5615419209934771s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2126s for a response
W20260807 11:10:54.290144 25981 connection.cc:232] RPC call timeout handler was delayed by -6.4733419208787382s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22240000000000004s for a response
W20260807 11:10:54.290494 25981 connection.cc:232] RPC call timeout handler was delayed by -6.3877727659419179s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20640000000000003s for a response
W20260807 11:10:54.290781 25981 connection.cc:232] RPC call timeout handler was delayed by -6.3625727659091353s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20920000000000002s for a response
I20260807 11:10:54.141161 29681 raft_consensus.cc:3032] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 2 LEADER]: Stepping down as leader of term 2
W20260807 11:10:54.291160 25981 connection.cc:232] RPC call timeout handler was delayed by -6.2779727659653872s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21860000000000002s for a response
I20260807 11:10:54.291327 29681 raft_consensus.cc:729] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:54.135442 25926 raft_consensus.cc:2454] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:54.292248 29681 consensus_queue.cc:260] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:54.293413 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:54.293594 29681 raft_consensus.cc:3037] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:54.294082 25926 raft_consensus.cc:2454] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:54.295014 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:54.206100 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:54.296005 25799 raft_consensus.cc:2396] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 6 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 11:10:54.296419 25799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1737) took 1673 ms. Trace:
I20260807 11:10:54.175336 26037 rpcz_store.cc:276] 0807 11:10:52.260389 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.116545 (+1856156us) service_pool.cc:227] Handling call
0807 11:10:54.125468 (+ 8923us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:54.174675 (+ 49207us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.174867 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.296702 25799 rpcz_store.cc:276] 0807 11:10:52.623206 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.205795 (+1582589us) service_pool.cc:227] Handling call
0807 11:10:54.296362 (+ 90567us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.297521 26037 raft_consensus.cc:1226] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:54.297746 25799 raft_consensus.cc:1226] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6. Ops: []
I20260807 11:10:54.298132 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 442) took 1949 ms. Trace:
I20260807 11:10:54.298379 25799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53324 (request call id 1551) took 1534 ms. Trace:
I20260807 11:10:54.298709 26976 raft_consensus.cc:2454] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:10:54.298799 25799 rpcz_store.cc:276] 0807 11:10:52.764321 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.297349 (+1533028us) service_pool.cc:227] Handling call
0807 11:10:54.297609 (+ 260us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:54.298132 (+ 523us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.298292 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.299417 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:54.300048 25920 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 8.
I20260807 11:10:54.300202 25799 raft_consensus.cc:1226] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:54.206697 29840 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:54.300567 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1835) took 2026 ms. Trace:
I20260807 11:10:54.300880 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:54.301186 26974 raft_consensus.cc:2454] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 5.
I20260807 11:10:54.301075 26039 rpcz_store.cc:276] 0807 11:10:52.273758 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.183302 (+1909544us) service_pool.cc:227] Handling call
0807 11:10:54.300062 (+116760us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:54.300266 (+ 204us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:54.300291 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:54.300329 (+ 38us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:54.300366 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.300422 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:54.300515 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.301635 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:54.302037 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:54.302426 26974 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50960 (request call id 367) took 1715 ms (client timeout 2131 ms). Trace:
I20260807 11:10:54.302733 26039 raft_consensus.cc:1226] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260807 11:10:54.302995 25920 raft_consensus.cc:2396] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260807 11:10:54.215688 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 103607us
W20260807 11:10:54.303684 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 121902us
I20260807 11:10:54.303995 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:54.266073 25872 leader_election.cc:302] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:54.304613 25925 raft_consensus.cc:2454] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:54.285686 26042 rpcz_store.cc:276] 0807 11:10:52.263020 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.172138 (+1909118us) service_pool.cc:227] Handling call
0807 11:10:54.172441 (+ 303us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:54.172594 (+ 153us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:54.172618 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:54.172654 (+ 36us) raft_consensus.cc:1631] Marking committed up to 8
0807 11:10:54.172682 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.172729 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:54.182689 (+ 9960us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.304432 25869 leader_election.cc:302] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:54.306828 25797 raft_consensus.cc:1226] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 5. Current term is 6. Ops: []
I20260807 11:10:54.285152 25800 rpcz_store.cc:276] 0807 11:10:52.620955 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.112974 (+1492019us) service_pool.cc:227] Handling call
0807 11:10:54.284655 (+171681us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.289987 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:54.307708 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1837) took 2028 ms. Trace:
I20260807 11:10:54.308516 25869 leader_election.cc:302] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:54.298482 26037 rpcz_store.cc:276] 0807 11:10:52.348460 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.297060 (+1948600us) service_pool.cc:227] Handling call
0807 11:10:54.297362 (+ 302us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:54.297903 (+ 541us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.298082 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.249541 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1823) took 2134 ms. Trace:
I20260807 11:10:54.166242 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 435) took 1903 ms. Trace:
I20260807 11:10:54.214058 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:54.299517 26976 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50960 (request call id 370) took 1599 ms (client timeout 2026 ms). Trace:
I20260807 11:10:54.220399 26033 raft_consensus.cc:2454] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:10:54.213744 29727 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:54.308075 26041 rpcz_store.cc:276] 0807 11:10:52.278949 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.183678 (+1904729us) service_pool.cc:227] Handling call
0807 11:10:54.211364 (+ 27686us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:54.211771 (+ 407us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:54.211802 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:54.212260 (+ 458us) log.cc:842] Serialized 26 byte log entry
0807 11:10:54.307381 (+ 95121us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:54.307422 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.307483 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:54.307528 (+ 45us) raft_consensus.cc:1671] finished
0807 11:10:54.307538 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:54.307640 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":29745,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":15317,"thread_start_us":94766,"threads_started":1,"wal-append.queue_time_us":65801}
I20260807 11:10:54.310627 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:54.310866 26034 rpcz_store.cc:276] 0807 11:10:52.262577 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.165282 (+1902705us) service_pool.cc:227] Handling call
0807 11:10:54.165646 (+ 364us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:54.165841 (+ 195us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:54.165876 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:54.165928 (+ 52us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:10:54.165977 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.166040 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:54.166173 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.311729 25916 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
W20260807 11:10:54.302839 26974 rpcz_store.cc:269] 0807 11:10:52.587311 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.059902 (+472591us) service_pool.cc:227] Handling call
0807 11:10:54.302247 (+1242345us) spinlock_profiling.cc:243] Waited 425 ms on lock 0x7b54005dcd54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:54.302370 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":977756544}
I20260807 11:10:54.359018 26035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 432) took 2098 ms. Trace:
I20260807 11:10:54.359313 26936 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:54.312813 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:54.360090 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 71217.6us
I20260807 11:10:54.241111 29720 raft_consensus.cc:482] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:54.363688 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:54.364279 26039 raft_consensus.cc:1261] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
W20260807 11:10:54.291492 25981 connection.cc:232] RPC call timeout handler was delayed by -5.6970560238696635s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15710000000000002s for a response
W20260807 11:10:54.365068 25981 connection.cc:232] RPC call timeout handler was delayed by -5.5971560238394886s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16820000000000002s for a response
W20260807 11:10:54.365458 25981 connection.cc:232] RPC call timeout handler was delayed by -4.9784381329081953s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30390000000000006s for a response
I20260807 11:10:54.308955 29712 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:54.367129 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 101184us
I20260807 11:10:54.359575 26035 rpcz_store.cc:276] 0807 11:10:52.260969 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.120580 (+1859611us) service_pool.cc:227] Handling call
0807 11:10:54.121625 (+ 1045us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:54.121818 (+ 193us) raft_consensus.cc:1520] Early marking committed up to index 6
0807 11:10:54.137188 (+ 15370us) log.cc:842] Serialized 14 byte log entry
0807 11:10:54.358568 (+221380us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:54.358619 (+ 51us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:10:54.358669 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.358840 (+ 171us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:54.358962 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":15676,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":3714,"thread_start_us":220658,"threads_started":1}
I20260807 11:10:54.305302 29654 raft_consensus.cc:482] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:54.289161 25808 consensus_queue.cc:260] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:54.318228 29608 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 11:10:54.315315 26976 rpcz_store.cc:269] 0807 11:10:52.700159 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.093164 (+393005us) service_pool.cc:227] Handling call
0807 11:10:54.299446 (+1206282us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.329738 25799 raft_consensus.cc:1226] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260807 11:10:54.346972 29840 raft_consensus.cc:2781] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 11:10:54.334759 25803 raft_consensus.cc:2454] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
W20260807 11:10:54.336282 25919 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1597) took 1651 ms (client timeout 1729 ms). Trace:
I20260807 11:10:54.340881 29813 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:54.310658 26046 rpcz_store.cc:276] 0807 11:10:52.115540 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.558585 (+443045us) service_pool.cc:227] Handling call
0807 11:10:52.558913 (+ 328us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:52.559112 (+ 199us) raft_consensus.cc:1520] Early marking committed up to index 5
0807 11:10:53.916084 (+1356972us) log.cc:842] Serialized 14 byte log entry
0807 11:10:54.249196 (+333112us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:54.249249 (+ 53us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:54.249310 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.249376 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:54.249485 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45294,"prepare.queue_time_us":1387774,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":28,"raft.queue_time_us":45450,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":16937,"thread_start_us":1643886,"threads_started":2,"wal-append.queue_time_us":287760}
W20260807 11:10:54.379347 25919 rpcz_store.cc:269] 0807 11:10:52.684778 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:52.897125 (+212347us) service_pool.cc:227] Handling call
0807 11:10:54.336046 (+1438921us) spinlock_profiling.cc:243] Waited 456 ms on lock 0x7b54003b29d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:54.336207 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1049425664}
W20260807 11:10:54.356739 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 48287.9us
I20260807 11:10:54.379585 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:54.380527 25920 raft_consensus.cc:2454] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 5.
I20260807 11:10:54.380594 25869 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:54.313054 25797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1724) took 1816 ms. Trace:
I20260807 11:10:54.381302 25920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1592) took 1799 ms. Trace:
I20260807 11:10:54.381682 25981 leader_election.cc:398] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 election: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6.
I20260807 11:10:54.381662 25920 rpcz_store.cc:276] 0807 11:10:52.581716 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.359101 (+1777385us) service_pool.cc:227] Handling call
0807 11:10:54.381244 (+ 22143us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.173123 26985 raft_consensus.cc:2396] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260807 11:10:54.383913 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:54.384186 25803 raft_consensus.cc:1226] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:54.384514 25919 raft_consensus.cc:2454] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:54.360450 29779 raft_consensus.cc:2726] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:54.360455 25801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1734) took 1738 ms. Trace:
I20260807 11:10:54.267686 29267 raft_consensus.cc:729] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:54.314961 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:54.389247 25921 raft_consensus.cc:3037] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:54.364434 26938 leader_election.cc:302] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc
W20260807 11:10:54.366694 25752 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.025s sys 0.029s
I20260807 11:10:54.317917 25796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1732) took 1696 ms. Trace:
I20260807 11:10:54.285108 29682 raft_consensus.cc:2726] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 11:10:54.284441 29609 raft_consensus.cc:2726] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:54.369649 25800 raft_consensus.cc:1226] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260807 11:10:54.374825 25808 raft_consensus.cc:3037] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:54.374115 29644 raft_consensus.cc:3037] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
W20260807 11:10:54.334609 25917 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35276 (request call id 287) took 1957 ms (client timeout 2538 ms). Trace:
I20260807 11:10:54.360875 26037 raft_consensus.cc:1226] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:54.361104 25750 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:54.402766 25917 rpcz_store.cc:269] 0807 11:10:52.377455 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.055313 (+1677858us) service_pool.cc:227] Handling call
0807 11:10:54.334336 (+279023us) spinlock_profiling.cc:243] Waited 88.2 ms on lock 0x7b5400408654. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:54.334526 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":202808064}
I20260807 11:10:54.404250 26936 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:54.382328 25981 leader_election.cc:401] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260807 11:10:54.381453 25797 rpcz_store.cc:276] 0807 11:10:52.497081 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.306286 (+1809205us) service_pool.cc:227] Handling call
0807 11:10:54.306553 (+ 267us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:54.312831 (+ 6278us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.313004 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.361187 25806 raft_consensus.cc:1261] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260807 11:10:54.306349 29323 consensus_queue.cc:1046] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.091s
I20260807 11:10:54.363366 29725 raft_consensus.cc:2726] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:54.385746 25869 leader_election.cc:302] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:54.386400 25801 rpcz_store.cc:276] 0807 11:10:52.622303 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.283323 (+1661020us) service_pool.cc:227] Handling call
0807 11:10:54.360394 (+ 77071us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.388669 25916 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35276 (request call id 288) took 2011 ms. Trace:
I20260807 11:10:54.459045 25916 rpcz_store.cc:276] 0807 11:10:52.377749 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.063767 (+1686018us) service_pool.cc:227] Handling call
0807 11:10:54.388509 (+324742us) spinlock_profiling.cc:243] Waited 55.7 ms on lock 0x7b54001df954. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:54.388606 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":128146048}
I20260807 11:10:54.390456 25796 rpcz_store.cc:276] 0807 11:10:52.621678 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.213798 (+1592120us) service_pool.cc:227] Handling call
0807 11:10:54.317845 (+104047us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:10:54.390434 25752 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.542s user 0.026s sys 0.029s
I20260807 11:10:54.363229 29750 consensus_queue.cc:1035] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20260807 11:10:54.368618 25872 leader_election.cc:302] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:54.369115 29720 raft_consensus.cc:3037] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:54.393878 25799 raft_consensus.cc:1226] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:54.397415 25803 raft_consensus.cc:3032] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:54.525171 25803 raft_consensus.cc:729] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:54.401398 29792 raft_consensus.cc:2781] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:10:54.367960 29839 raft_consensus.cc:2726] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:54.376106 29745 raft_consensus.cc:2781] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:10:54.402245 25931 raft_consensus.cc:2379] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 3: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:54.404367 25750 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:54.313691 29761 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:54.383518 29840 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:54.422039 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:54.313205 29417 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:54.451540 25808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1726) took 1916 ms. Trace:
I20260807 11:10:54.301321 29278 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:54.376820 29608 raft_consensus.cc:3037] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:54.454463 25800 raft_consensus.cc:1226] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:54.452610 25926 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:10:54.529498 29635 consensus_queue.cc:1046] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
W20260807 11:10:54.546240 29855 log.cc:924] Time spent T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20260807 11:10:54.547539 26036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1831) took 2356 ms. Trace:
I20260807 11:10:54.377378 29712 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:54.548033 26036 rpcz_store.cc:276] 0807 11:10:52.190862 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.848895 (+1658033us) service_pool.cc:227] Handling call
0807 11:10:53.893272 (+ 44377us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:53.893559 (+ 287us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:53.893595 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:53.923665 (+ 30070us) log.cc:842] Serialized 26 byte log entry
0807 11:10:54.281220 (+357555us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:54.281259 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.281338 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:54.547266 (+265928us) raft_consensus.cc:1671] finished
0807 11:10:54.547295 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:54.547467 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":324070,"threads_started":1,"wal-append.queue_time_us":433982}
I20260807 11:10:54.373315 29798 raft_consensus.cc:686] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:54.361387 29730 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:54.508446 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 281) took 2138 ms. Trace:
I20260807 11:10:54.374621 29732 consensus_queue.cc:1035] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.038s
I20260807 11:10:54.387791 29173 raft_consensus.cc:2781] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:54.374434 29715 consensus_queue.cc:1046] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260807 11:10:54.518728 25921 raft_consensus.cc:2454] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:10:54.521088 26985 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 431) took 1960 ms. Trace:
I20260807 11:10:54.561308 26985 rpcz_store.cc:276] 0807 11:10:52.560544 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.087411 (+1526867us) service_pool.cc:227] Handling call
0807 11:10:54.520863 (+433452us) spinlock_profiling.cc:243] Waited 52 ms on lock 0x7b54006edf54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:54.521025 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":119511936}
W20260807 11:10:54.523077 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 137395us
I20260807 11:10:54.562415 25921 raft_consensus.cc:1261] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 11:10:54.526161 25925 raft_consensus.cc:2454] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 2.
I20260807 11:10:54.374368 29696 consensus_queue.cc:1046] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.086s
I20260807 11:10:54.454916 29670 raft_consensus.cc:2726] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6.
W20260807 11:10:54.529870 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 148328us
I20260807 11:10:54.525981 25803 consensus_queue.cc:260] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:54.571852 25803 raft_consensus.cc:3037] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:54.533113 25916 raft_consensus.cc:1226] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 5. Current term is 6. Ops: []
I20260807 11:10:54.573554 25916 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 282) took 2197 ms. Trace:
I20260807 11:10:54.574002 25916 rpcz_store.cc:276] 0807 11:10:52.375957 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.459891 (+2083934us) service_pool.cc:227] Handling call
0807 11:10:54.532932 (+ 73041us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:54.573225 (+ 40293us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.573483 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.458071 29838 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:54.379115 29654 raft_consensus.cc:504] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:54.536547 25808 rpcz_store.cc:276] 0807 11:10:52.535510 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.287408 (+1751898us) service_pool.cc:227] Handling call
0807 11:10:54.287795 (+ 387us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:54.451126 (+163331us) raft_consensus.cc:1520] Early marking committed up to index 2
0807 11:10:54.451160 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:54.451215 (+ 55us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:54.451256 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.451328 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:54.451458 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":33982,"raft.run_cpu_time_us":13401,"raft.run_wall_time_us":73127}
I20260807 11:10:54.542138 25752 leader_election.cc:302] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:54.558900 25917 rpcz_store.cc:276] 0807 11:10:52.370379 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.449419 (+2079040us) service_pool.cc:227] Handling call
0807 11:10:54.507790 (+ 58371us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:54.508022 (+ 232us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:54.508057 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:54.508111 (+ 54us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:10:54.508160 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.508227 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:54.508366 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:10:54.526139 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 157597us
I20260807 11:10:54.527487 29857 raft_consensus.cc:482] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:54.563542 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 159224us
I20260807 11:10:54.537653 29840 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Advancing to term 9
I20260807 11:10:54.560514 29173 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 11:10:54.535943 29636 consensus_meta.cc:287] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.531s user 0.014s sys 0.003s
I20260807 11:10:54.527748 29776 raft_consensus.cc:482] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:54.563071 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1584) took 2006 ms. Trace:
I20260807 11:10:54.406317 25919 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59"
dest_uuid: "4fd6381544574ddca559ce63b9243efc"
mode: GRACEFUL
new_leader_uuid: "b22d69db45194908ad3d345eba9badd9"
from {username='slave'} at 127.0.0.1:34992
I20260807 11:10:54.598311 25919 raft_consensus.cc:595] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 5 LEADER]: Received request to transfer leadership to TS b22d69db45194908ad3d345eba9badd9
I20260807 11:10:54.581797 26984 raft_consensus.cc:2379] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 3: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:54.524852 29609 raft_consensus.cc:2726] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:54.532735 29847 raft_consensus.cc:2781] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260807 11:10:54.603670 29861 log.cc:924] Time spent T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.080s user 0.000s sys 0.001s
I20260807 11:10:54.603713 29847 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:54.604519 29847 raft_consensus.cc:3037] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:54.604914 29798 consensus_queue.cc:237] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:54.562605 25869 leader_election.cc:302] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:54.609702 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 47188.3us
I20260807 11:10:54.610781 25869 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:54.587992 25918 raft_consensus.cc:2379] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 8: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:54.588588 25917 raft_consensus.cc:1261] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 11:10:54.583801 25872 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:54.598724 25981 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:54.548041 29813 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:54.550403 29745 raft_consensus.cc:686] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:54.597700 25921 rpcz_store.cc:276] 0807 11:10:52.556872 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.561558 (+2004686us) service_pool.cc:227] Handling call
0807 11:10:54.561962 (+ 404us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:54.562840 (+ 878us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.563015 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.597400 25916 raft_consensus.cc:1261] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 11:10:54.615216 25786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53324 (request call id 1559) took 1127 ms. Trace:
I20260807 11:10:54.627704 25916 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1585) took 2070 ms. Trace:
I20260807 11:10:54.615842 25803 raft_consensus.cc:1261] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260807 11:10:54.628461 25916 rpcz_store.cc:276] 0807 11:10:52.557213 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.575008 (+2017795us) service_pool.cc:227] Handling call
0807 11:10:54.597002 (+ 21994us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:54.627362 (+ 30360us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.627629 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.629143 25803 pending_rounds.cc:76] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815: Aborting all ops after (but not including) 5
I20260807 11:10:54.629531 25803 pending_rounds.cc:97] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260807 11:10:54.629828 25803 raft_consensus.cc:2866] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:10:54.615592 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1586) took 2058 ms. Trace:
I20260807 11:10:54.630535 25917 rpcz_store.cc:276] 0807 11:10:52.557527 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.578074 (+2020547us) service_pool.cc:227] Handling call
0807 11:10:54.578444 (+ 370us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:54.615216 (+ 36772us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.615518 (+ 302us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.630836 25869 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:54.627494 25786 rpcz_store.cc:276] 0807 11:10:53.487708 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.302656 (+814948us) service_pool.cc:227] Handling call
0807 11:10:54.302933 (+ 277us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:54.303147 (+ 214us) raft_consensus.cc:1520] Early marking committed up to index 2
0807 11:10:54.378175 (+ 75028us) log.cc:842] Serialized 14 byte log entry
0807 11:10:54.614747 (+236572us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:54.614830 (+ 83us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:54.614906 (+ 76us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.614995 (+ 89us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:54.615146 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":61020,"thread_start_us":210259,"threads_started":2,"wal-append.queue_time_us":216048}
I20260807 11:10:54.637398 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:54.638585 26037 raft_consensus.cc:2454] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:10:54.636742 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:54.640214 26036 raft_consensus.cc:2379] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 6: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:54.642737 25919 raft_consensus.cc:1226] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:54.643455 25921 raft_consensus.cc:1226] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260807 11:10:54.637086 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:54.643100 25872 leader_election.cc:302] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:54.644719 26039 raft_consensus.cc:2379] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 3: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:54.600989 29173 raft_consensus.cc:3037] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:54.600656 29702 raft_consensus.cc:3037] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:54.659008 25919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1598) took 1959 ms. Trace:
I20260807 11:10:54.659591 25919 rpcz_store.cc:276] 0807 11:10:52.699532 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.638431 (+1938899us) service_pool.cc:227] Handling call
0807 11:10:54.642556 (+ 4125us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:54.658682 (+ 16126us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.658934 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:54.625150 29760 raft_consensus.cc:2726] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 11:10:54.534073 29267 consensus_queue.cc:260] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:54.710928 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 112283us
I20260807 11:10:54.715306 25981 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:54.711297 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:54.717232 26046 raft_consensus.cc:3037] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:54.623741 29669 raft_consensus.cc:2726] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260807 11:10:54.711992 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 129163us
I20260807 11:10:54.712163 25916 raft_consensus.cc:1226] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260807 11:10:54.622203 29725 raft_consensus.cc:2726] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:10:54.577636 29852 consensus_queue.cc:1046] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.102s
I20260807 11:10:54.636519 29779 raft_consensus.cc:2726] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:54.632968 29866 raft_consensus.cc:2726] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:54.727793 25750 leader_election.cc:302] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:54.653622 29728 raft_consensus.cc:2726] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:10:54.728936 25750 leader_election.cc:302] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:54.729180 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:54.731042 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:54.731802 25799 raft_consensus.cc:2454] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:54.612579 29636 raft_consensus.cc:504] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:54.740926 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:54.671504 29639 raft_consensus.cc:3032] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 11:10:54.715538 29715 raft_consensus.cc:3032] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:54.677713 29727 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:54.633469 29634 consensus_queue.cc:1035] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.090s
I20260807 11:10:54.757524 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:10:54.745519 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 203458us
I20260807 11:10:54.625560 29776 raft_consensus.cc:504] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:54.773357 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 44496.3us
I20260807 11:10:54.773765 25932 raft_consensus.cc:2379] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 5: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:54.729825 25803 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:54.774098 25750 leader_election.cc:302] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:54.598001 29857 raft_consensus.cc:504] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:54.726402 29761 raft_consensus.cc:2781] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:10:54.780409 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:54.724545 29205 raft_consensus.cc:2726] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:54.750406 25786 raft_consensus.cc:3032] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:10:54.781082 25800 raft_consensus.cc:3037] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:54.781352 25786 raft_consensus.cc:729] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:54.721025 29516 consensus_queue.cc:1046] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 3, Time since last communication: 0.035s
W20260807 11:10:54.769572 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 126553us
I20260807 11:10:54.755232 29624 raft_consensus.cc:2741] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
W20260807 11:10:54.772912 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 57733.2us
I20260807 11:10:54.778007 25916 raft_consensus.cc:1226] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 7. Current term is 8. Ops: []
I20260807 11:10:54.745293 29639 raft_consensus.cc:729] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:54.756929 28964 raft_consensus.cc:2781] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260807 11:10:54.765221 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 361039us
I20260807 11:10:54.790005 26046 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:54.791373 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:54.771798 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:54.815722 26036 raft_consensus.cc:1261] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 11:10:54.794991 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:54.748958 29715 raft_consensus.cc:729] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:54.797649 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:54.798547 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:54.825687 25916 raft_consensus.cc:1261] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260807 11:10:54.795133 29732 consensus_queue.cc:1035] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260807 11:10:54.834178 26936 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:54.780704 29761 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:54.796036 29589 raft_consensus.cc:2781] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:10:54.851974 26936 leader_election.cc:302] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:54.840427 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:54.853574 25916 raft_consensus.cc:1226] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 6. Current term is 7. Ops: []
I20260807 11:10:54.854043 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:54.821441 29516 consensus_queue.cc:1225] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer 4fd6381544574ddca559ce63b9243efc log is divergent from this leader: its last log entry 7.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260807 11:10:54.817783 28964 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:54.854489 25800 raft_consensus.cc:2396] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260807 11:10:54.846669 25786 consensus_queue.cc:260] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:54.839087 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:54.862320 29696 consensus_queue.cc:1046] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260807 11:10:54.889776 29589 raft_consensus.cc:686] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:54.849347 29730 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:54.934914 25981 leader_election.cc:302] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc
W20260807 11:10:54.917021 29868 log.cc:924] Time spent T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.194s user 0.000s sys 0.001s
I20260807 11:10:54.861963 29635 consensus_queue.cc:1046] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
I20260807 11:10:54.851109 29662 consensus_queue.cc:1035] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.059s
I20260807 11:10:54.919528 29692 raft_consensus.cc:2781] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:10:54.920629 29681 raft_consensus.cc:3032] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:54.855784 29875 consensus_queue.cc:1046] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.085s
I20260807 11:10:54.861521 28964 raft_consensus.cc:3037] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:54.856400 29725 consensus_queue.cc:1035] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.065s
I20260807 11:10:54.921775 29857 leader_election.cc:288] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:54.969524 29875 consensus_queue.cc:1046] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260807 11:10:54.970460 29875 consensus_queue.cc:1225] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer 4fd6381544574ddca559ce63b9243efc log is divergent from this leader: its last log entry 6.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260807 11:10:54.922062 29608 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:54.923125 25808 raft_consensus.cc:3032] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 11:10:54.979494 25808 raft_consensus.cc:729] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:54.924666 25806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53324 (request call id 1567) took 1070 ms. Trace:
I20260807 11:10:54.980536 25806 rpcz_store.cc:276] 0807 11:10:53.854212 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.450980 (+596768us) service_pool.cc:227] Handling call
0807 11:10:54.451341 (+ 361us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:54.519834 (+ 68493us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:54.519865 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:54.528477 (+ 8612us) log.cc:842] Serialized 26 byte log entry
0807 11:10:54.696094 (+167617us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:54.696160 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.696264 (+ 104us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:54.924467 (+228203us) raft_consensus.cc:1671] finished
0807 11:10:54.924485 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:54.924609 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":117208,"raft.queue_time_us":1114,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":5876,"thread_start_us":130442,"threads_started":1,"wal-append.queue_time_us":267210}
I20260807 11:10:54.927520 29702 raft_consensus.cc:2726] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:54.927680 25797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 355) took 1098 ms. Trace:
I20260807 11:10:54.850893 29761 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:54.927923 26984 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 462) took 1117 ms. Trace:
I20260807 11:10:54.997889 26984 rpcz_store.cc:276] 0807 11:10:53.810744 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.051175 (+240431us) service_pool.cc:227] Handling call
0807 11:10:54.927715 (+876540us) spinlock_profiling.cc:243] Waited 529 ms on lock 0x7b540068c254. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:54.927864 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1217204608}
I20260807 11:10:54.945051 29844 raft_consensus.cc:2741] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20260807 11:10:54.827214 29830 consensus_queue.cc:1035] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.074s
I20260807 11:10:54.930507 25800 raft_consensus.cc:1261] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 11:10:54.862237 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:55.023559 25796 raft_consensus.cc:3037] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:54.934065 25916 raft_consensus.cc:1261] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260807 11:10:54.936561 25932 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1603) took 2232 ms (client timeout 2156 ms). Trace:
I20260807 11:10:54.867193 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:55.026731 25803 raft_consensus.cc:2454] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:54.943349 25931 raft_consensus.cc:1226] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:54.950275 29681 raft_consensus.cc:729] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:55.028298 25921 raft_consensus.cc:1226] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260807 11:10:54.838987 29417 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:54.946096 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:54.782614 29745 consensus_queue.cc:237] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:54.949697 29636 leader_election.cc:288] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:54.968972 29776 leader_election.cc:288] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:54.967846 29173 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:54.922650 29876 raft_consensus.cc:2781] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader election won for term 3
W20260807 11:10:54.945248 29863 log.cc:924] Time spent T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.167s user 0.001s sys 0.001s
I20260807 11:10:54.970808 29840 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:54.984416 25797 rpcz_store.cc:276] 0807 11:10:53.830507 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.405074 (+574567us) service_pool.cc:227] Handling call
0807 11:10:54.450888 (+ 45814us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:54.451540 (+ 652us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:54.451582 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:54.466173 (+ 14591us) log.cc:842] Serialized 26 byte log entry
0807 11:10:54.752181 (+286008us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:10:54.752223 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.753079 (+ 856us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:54.927460 (+174381us) raft_consensus.cc:1671] finished
0807 11:10:54.927484 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:54.927624 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5950,"spinlock_wait_cycles":108175616,"thread_start_us":129539,"threads_started":1,"wal-append.queue_time_us":298744}
I20260807 11:10:54.928656 29639 consensus_queue.cc:260] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:55.028798 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 253435us
W20260807 11:10:55.025089 25932 rpcz_store.cc:269] 0807 11:10:52.703845 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.037111 (+1333266us) service_pool.cc:227] Handling call
0807 11:10:54.936302 (+899191us) spinlock_profiling.cc:243] Waited 735 ms on lock 0x7b54000532d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:54.936473 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1691313408}
I20260807 11:10:55.043332 26979 raft_consensus.cc:2454] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:10:54.998538 29760 raft_consensus.cc:2781] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:54.977916 29692 raft_consensus.cc:686] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
W20260807 11:10:55.051671 29836 log.cc:924] Time spent T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260807 11:10:55.061424 25869 leader_election.cc:302] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:54.984829 29323 consensus_queue.cc:1046] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.071s
I20260807 11:10:55.026988 25920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 285) took 2650 ms. Trace:
W20260807 11:10:55.114656 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 53370us
I20260807 11:10:55.116468 25920 rpcz_store.cc:276] 0807 11:10:52.376866 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.561665 (+2184799us) service_pool.cc:227] Handling call
0807 11:10:54.599770 (+ 38105us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:54.620987 (+ 21217us) raft_consensus.cc:1520] Early marking committed up to index 5
0807 11:10:54.654304 (+ 33317us) log.cc:842] Serialized 14 byte log entry
0807 11:10:55.026591 (+372287us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:55.026692 (+ 101us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:55.026763 (+ 71us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:55.026827 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:55.026938 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":31960,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":3758,"raft.queue_time_us":1418,"raft.run_cpu_time_us":329,"raft.run_wall_time_us":27880,"thread_start_us":281418,"threads_started":1}
I20260807 11:10:55.134470 25796 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:10:55.120040 25869 leader_election.cc:302] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:55.063211 29792 raft_consensus.cc:686] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:55.058446 29776 raft_consensus.cc:3032] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 11:10:55.028041 29702 consensus_queue.cc:1046] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.040s
I20260807 11:10:55.138274 25750 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:55.143368 25750 leader_election.cc:302] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:55.145181 29725 raft_consensus.cc:2726] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:55.146006 25750 leader_election.cc:302] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:55.145887 29765 consensus_queue.cc:1035] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:10:55.052095 29589 consensus_queue.cc:237] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:55.146127 29792 consensus_queue.cc:237] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:55.147116 29880 log.cc:924] Time spent T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20260807 11:10:55.147735 29765 raft_consensus.cc:2726] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:55.046722 29875 consensus_queue.cc:1035] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.067s
I20260807 11:10:55.067853 29876 raft_consensus.cc:686] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:55.141508 29776 raft_consensus.cc:729] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:55.114455 29639 raft_consensus.cc:3037] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
W20260807 11:10:55.170387 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 115596us
W20260807 11:10:55.165023 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 45063.9us
I20260807 11:10:55.107357 29445 consensus_queue.cc:1035] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.110s
I20260807 11:10:55.175467 29725 raft_consensus.cc:2726] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:55.284952 26041 raft_consensus.cc:1226] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:55.303085 25869 leader_election.cc:302] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:55.157541 29765 raft_consensus.cc:2726] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260807 11:10:55.233722 29657 consensus_meta.cc:287] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 1.161s user 0.014s sys 0.000s
I20260807 11:10:55.153126 29799 consensus_queue.cc:1035] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.139s
I20260807 11:10:55.221196 29278 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:55.283915 29778 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.512s user 0.004s sys 0.046s
I20260807 11:10:55.305522 25981 leader_election.cc:398] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6.
I20260807 11:10:55.332803 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1832) took 3140 ms. Trace:
I20260807 11:10:55.338157 26038 rpcz_store.cc:276] 0807 11:10:52.192792 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.113798 (+1921006us) service_pool.cc:227] Handling call
0807 11:10:54.114201 (+ 403us) raft_consensus.cc:1457] Updating replica for 3 ops
0807 11:10:54.205784 (+ 91583us) raft_consensus.cc:1520] Early marking committed up to index 1
0807 11:10:54.205813 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 11:10:54.206380 (+ 567us) log.cc:842] Serialized 52 byte log entry
0807 11:10:54.370279 (+163899us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:10:54.370315 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.370373 (+ 58us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:55.332482 (+962109us) raft_consensus.cc:1671] finished
0807 11:10:55.332512 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:55.332698 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":42502,"raft.queue_time_us":68270,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":94,"thread_start_us":163543,"threads_started":1,"wal-append.queue_time_us":465399}
I20260807 11:10:55.138499 29681 consensus_queue.cc:260] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:55.310990 25869 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:55.349648 26980 raft_consensus.cc:1226] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 LEADER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 5. Current term is 6. Ops: []
W20260807 11:10:55.350940 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 60976.7us
W20260807 11:10:55.339174 29838 consensus_meta.cc:287] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.752s user 0.005s sys 0.005s
I20260807 11:10:55.360633 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:10:55.342007 29847 consensus_meta.cc:287] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.737s user 0.004s sys 0.008s
I20260807 11:10:55.388561 25981 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:55.344499 29608 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:55.357082 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 46169.3us
W20260807 11:10:55.341957 29720 consensus_meta.cc:287] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.808s user 0.007s sys 0.004s
I20260807 11:10:55.387781 26042 raft_consensus.cc:3037] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:55.338631 29898 raft_consensus.cc:2781] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:55.405125 25797 raft_consensus.cc:1226] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:55.406658 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:55.407430 25797 raft_consensus.cc:3037] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:55.401995 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:55.354667 29730 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:55.360903 29609 raft_consensus.cc:2726] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 11:10:55.342213 29765 consensus_queue.cc:1046] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
W20260807 11:10:55.390347 29891 log.cc:924] Time spent T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.252s user 0.001s sys 0.000s
I20260807 11:10:55.422612 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:55.423595 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:55.417752 25797 raft_consensus.cc:2454] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:10:55.421388 25869 leader_election.cc:302] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
W20260807 11:10:55.435290 26939 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.015s sys 0.032s
I20260807 11:10:55.392508 29692 consensus_queue.cc:237] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:55.439121 29644 consensus_meta.cc:287] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.989s user 0.010s sys 0.000s
I20260807 11:10:55.397670 29691 raft_consensus.cc:2741] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20260807 11:10:55.446174 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:55.431339 29445 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:55.493618 26038 raft_consensus.cc:3037] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
W20260807 11:10:55.450801 26939 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.778s user 0.015s sys 0.033s
I20260807 11:10:55.391737 29838 raft_consensus.cc:2726] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260807 11:10:55.460064 26938 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.035s sys 0.051s
W20260807 11:10:55.500535 26938 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.553s user 0.035s sys 0.052s
W20260807 11:10:55.500420 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 54071us
W20260807 11:10:55.501303 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 554247us
I20260807 11:10:55.439455 29730 raft_consensus.cc:504] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:55.412925 29898 raft_consensus.cc:482] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:55.484833 29682 raft_consensus.cc:3032] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 LEADER]: Stepping down as leader of term 7
W20260807 11:10:55.485723 25798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1549) took 2777 ms (client timeout 2441 ms). Trace:
W20260807 11:10:55.529522 25798 rpcz_store.cc:269] 0807 11:10:52.708276 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:53.946469 (+1238193us) service_pool.cc:227] Handling call
0807 11:10:55.485496 (+1539027us) spinlock_profiling.cc:243] Waited 818 ms on lock 0x7b54003abbd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:55.485649 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1882056448}
I20260807 11:10:55.419163 29173 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:55.329303 29887 consensus_queue.cc:1046] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.099s
I20260807 11:10:55.430367 29681 raft_consensus.cc:3037] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:55.488811 25869 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:55.496802 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:55.393918 29847 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:55.460367 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:55.515631 25981 leader_election.cc:398] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 election: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6.
I20260807 11:10:55.515746 25800 raft_consensus.cc:1226] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 6. Current term is 7. Ops: []
I20260807 11:10:55.546271 25981 leader_election.cc:401] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260807 11:10:55.488298 29609 raft_consensus.cc:2726] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:10:55.355571 29776 consensus_queue.cc:260] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:55.507718 29636 raft_consensus.cc:2726] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:55.461035 29840 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 9 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:55.541035 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:55.547117 25796 raft_consensus.cc:3037] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
W20260807 11:10:55.555876 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 53147.1us
I20260807 11:10:55.413748 29720 raft_consensus.cc:504] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:55.549024 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 87317.5us
I20260807 11:10:55.528566 29898 raft_consensus.cc:3037] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:55.553162 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1738) took 1675 ms. Trace:
I20260807 11:10:55.565797 25925 rpcz_store.cc:276] 0807 11:10:53.877891 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.642880 (+764989us) service_pool.cc:227] Handling call
0807 11:10:54.721054 (+ 78174us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:54.721287 (+ 233us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:54.721311 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:54.721763 (+ 452us) log.cc:842] Serialized 26 byte log entry
0807 11:10:54.857325 (+135562us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:54.857378 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.857469 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:55.552871 (+695402us) raft_consensus.cc:1671] finished
0807 11:10:55.552899 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:55.553079 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":134445,"threads_started":1,"wal-append.queue_time_us":327480}
I20260807 11:10:55.549082 26035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 450) took 1366 ms. Trace:
I20260807 11:10:55.567433 26035 rpcz_store.cc:276] 0807 11:10:54.182236 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.368474 (+186238us) service_pool.cc:227] Handling call
0807 11:10:54.368884 (+ 410us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:54.384851 (+ 15967us) raft_consensus.cc:1520] Early marking committed up to index 6
0807 11:10:54.550206 (+165355us) spinlock_profiling.cc:243] Waited 8.31 ms on lock 0x7b4800130758. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc456b7f1 00007f3fc45628e7 00007f3fc456277a 00007f3fc465695f 00007f3fc4656627 00007f3fc466ab2c 00007f3fc466aad8 00007f3fc466aa58 00007f3fc466aa13 00007f3fc4669cbc 00007f3fc35ebbe6
0807 11:10:54.634128 (+ 83922us) log.cc:842] Serialized 14 byte log entry
0807 11:10:54.936947 (+302819us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:54.944399 (+ 7452us) log.cc:842] Serialized 26 byte log entry
0807 11:10:54.944593 (+ 194us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:10:54.944638 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.948820 (+ 4182us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:55.535485 (+586665us) raft_consensus.cc:1671] finished
0807 11:10:55.548824 (+ 13339us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:55.549004 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18633,"prepare.queue_time_us":161215,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":47,"raft.queue_time_us":7475,"raft.run_cpu_time_us":2560,"raft.run_wall_time_us":177522,"spinlock_wait_cycles":19113728,"thread_start_us":389082,"threads_started":2,"wal-append.queue_time_us":651983}
I20260807 11:10:55.569511 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:55.569996 26035 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Advancing to term 9
I20260807 11:10:55.572253 25981 leader_election.cc:302] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:55.529363 29682 raft_consensus.cc:729] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:55.562896 25869 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:55.541821 29838 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:55.590713 25869 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:55.598508 25869 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:55.565411 26038 raft_consensus.cc:2396] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 11:10:55.508016 29445 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:55.602272 25869 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:55.609644 25919 raft_consensus.cc:1226] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 5. Current term is 6. Ops: []
I20260807 11:10:55.560475 29173 raft_consensus.cc:2781] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:55.615556 25869 leader_election.cc:302] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:55.394102 29876 consensus_queue.cc:237] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:55.617324 26035 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 9.
I20260807 11:10:55.628677 25796 raft_consensus.cc:2454] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:55.629303 26938 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:10:55.623522 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 472522us
I20260807 11:10:55.621407 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:55.630533 25750 leader_election.cc:302] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:55.566584 29691 raft_consensus.cc:2726] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6.
I20260807 11:10:55.630789 26038 raft_consensus.cc:2379] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 3: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
W20260807 11:10:55.633864 26936 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.700s user 0.003s sys 0.015s
W20260807 11:10:55.634239 26936 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.701s user 0.003s sys 0.016s
I20260807 11:10:55.596892 29609 raft_consensus.cc:2726] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:10:55.644191 26035 raft_consensus.cc:1261] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 11:10:55.648926 25750 leader_election.cc:302] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:55.650820 25869 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:55.656808 25750 leader_election.cc:302] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:55.616155 29725 raft_consensus.cc:2781] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election won for term 3
W20260807 11:10:55.634564 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 701207us
I20260807 11:10:55.628579 29893 raft_consensus.cc:2781] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:55.624691 29173 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:55.608901 29644 consensus_queue.cc:1035] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.056s
W20260807 11:10:55.722769 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 150588us
I20260807 11:10:55.724395 26035 raft_consensus.cc:1226] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:55.653378 29589 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:55.608454 29838 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:55.723664 25981 leader_election.cc:302] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:10:55.727083 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 76396.4us
I20260807 11:10:55.747539 29792 raft_consensus.cc:482] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:55.751571 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 94884.7us
I20260807 11:10:55.660445 29634 raft_consensus.cc:2781] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:10:55.667423 29691 raft_consensus.cc:2781] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:55.754645 25869 leader_election.cc:302] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:55.701144 29893 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:55.721870 29725 raft_consensus.cc:686] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 3 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:55.615928 29669 consensus_queue.cc:1046] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
I20260807 11:10:55.674955 29609 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:55.719157 29173 raft_consensus.cc:3037] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:55.722368 29662 consensus_queue.cc:1035] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.079s
I20260807 11:10:55.727469 29802 raft_consensus.cc:2726] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 11:10:55.767824 29691 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:55.775086 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:55.775164 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:55.776980 25798 raft_consensus.cc:1261] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 11:10:55.777029 25869 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:55.756754 25750 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
W20260807 11:10:55.843145 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 66187us
W20260807 11:10:55.843518 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 86851.4us
I20260807 11:10:55.844563 25750 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:55.678102 29888 raft_consensus.cc:2781] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 FOLLOWER]: Leader election won for term 9
I20260807 11:10:55.779207 29893 raft_consensus.cc:3037] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:55.838781 29669 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:55.846019 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:55.845562 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:55.756286 29589 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:55.846036 25750 leader_election.cc:302] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:55.764678 29887 raft_consensus.cc:2726] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 11:10:56.021875 29445 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:55.856868 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:55.670006 29682 consensus_queue.cc:260] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:56.023816 29682 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:10:56.027176 29949 raft_consensus.cc:482] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:55.888157 29929 log.cc:924] Time spent T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.185s user 0.000s sys 0.001s
W20260807 11:10:55.889021 29928 log.cc:924] Time spent T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.238s user 0.001s sys 0.000s
I20260807 11:10:55.758220 29792 raft_consensus.cc:504] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:55.995417 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:55.855355 29680 raft_consensus.cc:2781] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:55.846211 29691 raft_consensus.cc:3037] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:55.873106 29888 raft_consensus.cc:686] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
W20260807 11:10:55.866870 29896 log.cc:924] Time spent T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.474s user 0.001s sys 0.000s
W20260807 11:10:55.855098 29902 log.cc:924] Time spent T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.299s user 0.001s sys 0.001s
I20260807 11:10:55.866786 29947 raft_consensus.cc:2781] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:55.774268 29278 consensus_queue.cc:1046] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.072s
I20260807 11:10:55.889626 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1732) took 1256 ms. Trace:
I20260807 11:10:56.093232 26034 rpcz_store.cc:276] 0807 11:10:54.633188 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.633373 (+ 185us) service_pool.cc:227] Handling call
0807 11:10:54.675189 (+ 41816us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:10:54.715913 (+ 40724us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:54.715944 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 11:10:54.716568 (+ 624us) log.cc:842] Serialized 52 byte log entry
0807 11:10:54.947251 (+230683us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:54.947285 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.947362 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:55.889359 (+941997us) raft_consensus.cc:1671] finished
0807 11:10:55.889386 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:55.889557 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":35340,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"thread_start_us":192137,"threads_started":1,"wal-append.queue_time_us":1087371}
W20260807 11:10:56.011145 26042 consensus_meta.cc:287] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.616s user 0.010s sys 0.014s
W20260807 11:10:56.011384 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 165396us
I20260807 11:10:56.095395 26042 raft_consensus.cc:2454] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
W20260807 11:10:56.012326 29745 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.008s sys 0.001s
I20260807 11:10:56.100632 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:56.022917 25806 raft_consensus.cc:2454] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:10:55.828436 29609 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:55.860775 29730 leader_election.cc:288] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:56.056969 29680 raft_consensus.cc:482] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:56.047305 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:56.123092 26035 raft_consensus.cc:3037] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
W20260807 11:10:55.885190 29922 log.cc:924] Time spent T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.234s user 0.001s sys 0.000s
I20260807 11:10:56.128696 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:56.063349 25798 raft_consensus.cc:2379] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 3: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
W20260807 11:10:56.065871 29923 log.cc:924] Time spent T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.550s user 0.008s sys 0.005s
I20260807 11:10:56.075959 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:56.080492 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:56.231954 26038 raft_consensus.cc:2396] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 7 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 11:10:56.232462 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:56.233328 25932 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Advancing to term 9
I20260807 11:10:56.001305 29915 raft_consensus.cc:2781] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election won for term 6
W20260807 11:10:56.047163 29924 log.cc:924] Time spent T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.333s user 0.006s sys 0.006s
I20260807 11:10:56.041277 29887 raft_consensus.cc:482] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:56.237939 29887 raft_consensus.cc:504] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:56.240234 29887 leader_election.cc:288] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:56.028023 29949 raft_consensus.cc:504] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:56.242605 29949 leader_election.cc:288] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:56.243256 29949 raft_consensus.cc:482] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:56.243788 29949 raft_consensus.cc:504] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:56.045267 29445 raft_consensus.cc:2726] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 11:10:56.105042 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:56.064469 29634 raft_consensus.cc:686] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:55.995266 29950 raft_consensus.cc:2781] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader election won for term 3
W20260807 11:10:56.098819 29938 log.cc:924] Time spent T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.314s user 0.007s sys 0.000s
W20260807 11:10:56.100780 29745 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.661s user 0.008s sys 0.002s
W20260807 11:10:56.282313 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 186350us
I20260807 11:10:56.288132 26979 raft_consensus.cc:1226] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 LEADER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260807 11:10:56.180310 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:56.001526 29669 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:56.245481 25920 raft_consensus.cc:2454] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 5.
W20260807 11:10:56.245708 25981 connection.cc:232] RPC call timeout handler was delayed by -1.8539684130810201s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1505s for a response
W20260807 11:10:56.297305 25981 connection.cc:232] RPC call timeout handler was delayed by -1.8116684129927307s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1552s for a response
W20260807 11:10:56.297704 25981 connection.cc:232] RPC call timeout handler was delayed by -1.7522684130817652s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16180000000000003s for a response
W20260807 11:10:56.298102 25981 connection.cc:232] RPC call timeout handler was delayed by -1.7162684130016714s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1658s for a response
W20260807 11:10:56.298520 25981 connection.cc:232] RPC call timeout handler was delayed by -1.6640684129670262s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1716s for a response
W20260807 11:10:56.298803 25981 connection.cc:232] RPC call timeout handler was delayed by -1.6640684129670262s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1716s for a response
W20260807 11:10:56.299086 25981 connection.cc:232] RPC call timeout handler was delayed by -1.6523684130515903s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17290000000000003s for a response
W20260807 11:10:56.299458 25981 connection.cc:232] RPC call timeout handler was delayed by -1.6523684130515903s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17290000000000003s for a response
W20260807 11:10:56.299816 25981 connection.cc:232] RPC call timeout handler was delayed by -1.6451684129424393s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17370000000000002s for a response
W20260807 11:10:56.300165 25981 connection.cc:232] RPC call timeout handler was delayed by -1.6451684129424393s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17370000000000002s for a response
W20260807 11:10:56.246148 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 163601us
I20260807 11:10:56.247022 25932 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 9.
I20260807 11:10:56.254565 29949 leader_election.cc:288] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:56.262490 25926 raft_consensus.cc:2379] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 5: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:56.130002 29680 raft_consensus.cc:3037] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
W20260807 11:10:56.300529 25981 connection.cc:232] RPC call timeout handler was delayed by -1.577668413054198s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18120000000000003s for a response
I20260807 11:10:56.045972 29725 consensus_queue.cc:237] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:56.642453 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1739) took 1715 ms. Trace:
W20260807 11:10:56.312428 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 262551us
I20260807 11:10:56.647171 26046 rpcz_store.cc:276] 0807 11:10:54.926574 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.926757 (+ 183us) service_pool.cc:227] Handling call
0807 11:10:54.927121 (+ 364us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:54.927505 (+ 384us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:54.927530 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:54.927980 (+ 450us) log.cc:842] Serialized 26 byte log entry
0807 11:10:55.229737 (+301757us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:55.229792 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:55.229890 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:56.641979 (+1412089us) raft_consensus.cc:1671] finished
0807 11:10:56.642247 (+ 268us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:56.642389 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1062,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":87,"thread_start_us":301327,"threads_started":1,"wal-append.queue_time_us":464022}
I20260807 11:10:56.414494 29792 leader_election.cc:288] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:56.413738 29609 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:56.643860 26036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1738) took 1717 ms. Trace:
I20260807 11:10:56.147470 29957 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:56.411139 29173 consensus_meta.cc:287] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.571s user 0.007s sys 0.003s
I20260807 11:10:56.261938 29852 raft_consensus.cc:2726] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260807 11:10:56.412158 29870 log.cc:924] Time spent T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 1.492s user 0.000s sys 0.017s
I20260807 11:10:56.400518 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:56.400102 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:56.420473 29857 raft_consensus.cc:2781] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 11:10:56.206034 29838 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:56.328379 26035 raft_consensus.cc:2454] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:56.649359 26936 leader_election.cc:302] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:56.262116 29901 raft_consensus.cc:2741] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260807 11:10:56.658156 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:56.416070 29720 leader_election.cc:288] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:56.661974 26036 rpcz_store.cc:276] 0807 11:10:54.926171 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.929265 (+ 3094us) service_pool.cc:227] Handling call
0807 11:10:54.929592 (+ 327us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:54.929848 (+ 256us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:54.929876 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:54.930276 (+ 400us) log.cc:842] Serialized 26 byte log entry
0807 11:10:55.069074 (+138798us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:10:55.069126 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:55.069211 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:56.634045 (+1564834us) raft_consensus.cc:1671] finished
0807 11:10:56.643682 (+ 9637us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:56.643806 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":138413,"threads_started":1,"wal-append.queue_time_us":657876}
W20260807 11:10:56.417599 29639 consensus_meta.cc:287] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 1.116s user 0.012s sys 0.000s
W20260807 11:10:56.641446 25981 connection.cc:232] RPC call timeout handler was delayed by -1.5632684130687267s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18280000000000002s for a response
I20260807 11:10:56.271811 29730 raft_consensus.cc:2781] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election won for term 4
W20260807 11:10:56.417716 29938 log.cc:924] Time spent T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
W20260807 11:10:56.417636 29932 log.cc:924] Time spent T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
W20260807 11:10:56.410640 29934 log.cc:924] Time spent T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.404s user 0.000s sys 0.000s
W20260807 11:10:56.410036 29900 log.cc:924] Time spent T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.546s user 0.000s sys 0.001s
I20260807 11:10:56.676007 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:56.685245 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:56.871462 26046 raft_consensus.cc:2454] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:56.686249 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:10:56.411454 29962 log.cc:924] Time spent T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.117s user 0.000s sys 0.000s
I20260807 11:10:56.306063 29950 raft_consensus.cc:686] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:56.409271 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:10:56.779245 29682 consensus_meta.cc:287] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.755s user 0.004s sys 0.008s
I20260807 11:10:56.707479 29712 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:56.686852 29957 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:56.709565 29608 raft_consensus.cc:3032] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:56.709730 29991 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:56.689597 29792 consensus_queue.cc:1035] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.032s
I20260807 11:10:56.685190 29173 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:56.359304 29888 consensus_queue.cc:237] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:56.941733 29608 raft_consensus.cc:729] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
W20260807 11:10:56.713846 29893 consensus_meta.cc:287] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.823s user 0.008s sys 0.001s
I20260807 11:10:56.714061 29972 raft_consensus.cc:482] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:56.716620 26984 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50942 (request call id 443) took 1279 ms. Trace:
I20260807 11:10:56.717478 29953 raft_consensus.cc:482] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
W20260807 11:10:56.779078 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 130334us
W20260807 11:10:56.736493 28964 consensus_meta.cc:287] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 1.760s user 0.013s sys 0.001s
W20260807 11:10:56.720722 29898 consensus_meta.cc:287] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 1.104s user 0.008s sys 0.000s
W20260807 11:10:56.736001 29931 log.cc:924] Time spent T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 1.178s user 0.007s sys 0.001s
W20260807 11:10:56.736960 29681 consensus_meta.cc:287] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 1.188s user 0.008s sys 0.004s
I20260807 11:10:56.958622 29608 consensus_queue.cc:260] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:56.732188 29881 raft_consensus.cc:482] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:56.731750 29847 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:56.360436 29883 raft_consensus.cc:482] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:56.696130 29589 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:56.906626 29761 consensus_meta.cc:287] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 1.878s user 0.015s sys 0.000s
I20260807 11:10:56.907035 29955 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:56.750389 29857 raft_consensus.cc:482] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:56.687639 29634 consensus_queue.cc:237] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:56.778582 29217 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:56.906636 29623 raft_consensus.cc:2726] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:56.663478 29949 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:56.907081 29680 raft_consensus.cc:504] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:56.941088 29730 raft_consensus.cc:686] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:56.909483 29950 consensus_queue.cc:237] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:56.738962 25752 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.963s user 0.021s sys 0.034s
I20260807 11:10:56.685497 25799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1747) took 1950 ms. Trace:
I20260807 11:10:56.714502 25918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 291) took 2755 ms. Trace:
I20260807 11:10:56.653494 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1736) took 2810 ms. Trace:
I20260807 11:10:56.706763 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:56.729732 25931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1737) took 2854 ms. Trace:
I20260807 11:10:56.749671 25932 raft_consensus.cc:2454] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
W20260807 11:10:56.774710 25981 connection.cc:232] RPC call timeout handler was delayed by -1.5632684130687267s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18280000000000002s for a response
I20260807 11:10:57.031356 29669 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:57.677629 25803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 365) took 1818 ms. Trace:
I20260807 11:10:57.837464 29589 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
W20260807 11:10:57.166852 26937 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.009s sys 0.004s
W20260807 11:10:57.840587 26937 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.194s user 0.013s sys 0.008s
I20260807 11:10:57.842080 29217 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:57.843924 25752 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.068s user 0.030s sys 0.038s
W20260807 11:10:57.727821 26042 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.136s user 0.007s sys 0.011s
W20260807 11:10:57.657372 25750 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.011s user 0.002s sys 0.026s
I20260807 11:10:57.759481 30029 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:57.316140 29445 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.020s sys 0.039s
I20260807 11:10:57.251739 29608 raft_consensus.cc:3037] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
W20260807 11:10:57.573864 29691 consensus_meta.cc:287] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 1.485s user 0.009s sys 0.002s
I20260807 11:10:57.851006 29680 leader_election.cc:288] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:57.841951 25981 connection.cc:232] RPC call timeout handler was delayed by -1.4957684129476547s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19030000000000002s for a response
W20260807 11:10:57.760092 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 982772us
I20260807 11:10:57.862095 29669 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:57.863263 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:57.870306 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:57.680398 25916 raft_consensus.cc:1226] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:57.837879 25803 rpcz_store.cc:276] 0807 11:10:55.858775 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:55.996966 (+138191us) service_pool.cc:227] Handling call
0807 11:10:55.997406 (+ 440us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:56.006956 (+ 9550us) raft_consensus.cc:1520] Early marking committed up to index 5
0807 11:10:56.057666 (+ 50710us) log.cc:842] Serialized 14 byte log entry
0807 11:10:56.367622 (+309956us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:56.368244 (+ 622us) log.cc:842] Serialized 26 byte log entry
0807 11:10:56.368412 (+ 168us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:56.368462 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:56.368560 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:57.674863 (+1306303us) raft_consensus.cc:1671] finished
0807 11:10:57.674978 (+ 115us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:57.677047 (+ 2069us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":50559,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":9262,"raft.queue_time_us":472636,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":173,"thread_start_us":264497,"threads_started":1,"wal-append.queue_time_us":1066928}
I20260807 11:10:57.137219 26979 raft_consensus.cc:2454] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 8.
I20260807 11:10:57.771205 26936 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:57.726660 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1726) took 3747 ms. Trace:
I20260807 11:10:57.005666 28964 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:57.912696 26042 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.321s user 0.009s sys 0.020s
W20260807 11:10:57.915213 25981 connection.cc:232] RPC call timeout handler was delayed by -1.4957684129476547s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19030000000000002s for a response
W20260807 11:10:57.922751 25981 connection.cc:232] RPC call timeout handler was delayed by -1.4858684130012989s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19140000000000001s for a response
W20260807 11:10:57.910284 25750 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.264s user 0.004s sys 0.036s
I20260807 11:10:56.989161 26984 rpcz_store.cc:276] 0807 11:10:55.437546 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:55.441434 (+ 3888us) service_pool.cc:227] Handling call
0807 11:10:55.441863 (+ 429us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:55.442331 (+ 468us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:55.442374 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:55.455409 (+ 13035us) log.cc:842] Serialized 26 byte log entry
0807 11:10:55.599966 (+144557us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:55.600027 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:55.600138 (+ 111us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:56.716383 (+1116245us) raft_consensus.cc:1671] finished
0807 11:10:56.716410 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:56.716553 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4855,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":13553,"thread_start_us":70169,"threads_started":1,"wal-append.queue_time_us":1214355}
W20260807 11:10:57.945672 25981 connection.cc:232] RPC call timeout handler was delayed by -1.4858684130012989s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19140000000000001s for a response
I20260807 11:10:57.839126 29589 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:57.010556 29802 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.951s user 0.011s sys 0.014s
W20260807 11:10:57.934012 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 3.15025e+06us
W20260807 11:10:57.952039 25981 connection.cc:232] RPC call timeout handler was delayed by -1.4750684129539877s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1926s for a response
W20260807 11:10:56.987317 29915 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.004s sys 0.021s
W20260807 11:10:57.319025 29963 log.cc:924] Time spent T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.656s user 0.001s sys 0.000s
W20260807 11:10:57.575959 29876 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.835s user 0.016s sys 0.013s
W20260807 11:10:57.373950 29958 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.644s user 0.013s sys 0.017s
W20260807 11:10:57.961982 25981 connection.cc:232] RPC call timeout handler was delayed by -1.4750684129539877s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1926s for a response
W20260807 11:10:57.568094 29890 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.826s user 0.004s sys 0.004s
W20260807 11:10:57.965636 29445 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.188s user 0.048s sys 0.080s
I20260807 11:10:57.965747 29857 raft_consensus.cc:3037] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
W20260807 11:10:57.967151 25981 connection.cc:232] RPC call timeout handler was delayed by -1.4732684129849076s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1928s for a response
I20260807 11:10:57.972811 29730 consensus_queue.cc:237] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:57.973237 29761 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:57.975780 25797 raft_consensus.cc:2379] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 5: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:57.975888 29761 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:57.977057 25981 connection.cc:232] RPC call timeout handler was delayed by -1.4732684129849076s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1928s for a response
I20260807 11:10:57.978626 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1751) took 1314 ms. Trace:
I20260807 11:10:57.014714 29972 raft_consensus.cc:504] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:57.979236 25932 rpcz_store.cc:276] 0807 11:10:56.645867 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:56.657815 (+ 11948us) service_pool.cc:227] Handling call
0807 11:10:57.954609 (+1296794us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:57.979065 29949 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:57.981810 29972 leader_election.cc:288] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:57.971808 30029 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:57.996979 29958 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.267s user 0.017s sys 0.023s
I20260807 11:10:57.982769 29949 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:57.982847 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:10:57.983837 29890 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.296s user 0.006s sys 0.006s
I20260807 11:10:56.985530 29893 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:57.984422 26038 raft_consensus.cc:2454] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
W20260807 11:10:57.987107 29802 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.918s user 0.013s sys 0.014s
I20260807 11:10:57.988129 29955 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:57.013597 29898 raft_consensus.cc:504] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:57.888805 25799 rpcz_store.cc:276] 0807 11:10:54.734605 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.738841 (+ 4236us) service_pool.cc:227] Handling call
0807 11:10:54.757734 (+ 18893us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:10:54.758185 (+ 451us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:54.758224 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 11:10:54.773847 (+ 15623us) log.cc:842] Serialized 52 byte log entry
0807 11:10:54.927056 (+153209us) spinlock_profiling.cc:243] Waited 1.94 ms on lock 0x7b1800263080. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbc77e345 00007f3fc55b5a39 00007f3fbc79b8bb 00007f3fbc799cbf 00007f3fbc79be22 00007f3fbf3455bb 00007f3fbf34b063 00007f3fbf34b411 00007f3fbf6985cf
0807 11:10:55.064164 (+137108us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:55.064218 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:55.064311 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:56.670495 (+1606184us) raft_consensus.cc:1671] finished
0807 11:10:56.670523 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:56.685417 (+ 14894us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21362,"raft.queue_time_us":14829,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":7757,"spinlock_wait_cycles":4453248,"thread_start_us":136854,"threads_started":1,"wal-append.queue_time_us":774113}
I20260807 11:10:57.989670 29883 raft_consensus.cc:504] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:57.992137 28964 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:57.965230 25931 rpcz_store.cc:276] 0807 11:10:53.876043 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.636185 (+760142us) service_pool.cc:227] Handling call
0807 11:10:54.937055 (+300870us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:55.027782 (+ 90727us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:56.729488 (+1701706us) spinlock_profiling.cc:243] Waited 300 ms on lock 0x7b54000532d0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:56.729667 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":690980608}
I20260807 11:10:58.029707 25926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35276 (request call id 301) took 1953 ms. Trace:
I20260807 11:10:58.030207 25926 rpcz_store.cc:276] 0807 11:10:56.075860 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:56.076630 (+ 770us) service_pool.cc:227] Handling call
0807 11:10:58.029243 (+1952613us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b5400035cd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:58.029515 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":332418816}
I20260807 11:10:58.031236 29883 leader_election.cc:288] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:58.031363 25931 raft_consensus.cc:1226] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7. Ops: []
I20260807 11:10:58.031993 29883 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:58.032262 25926 raft_consensus.cc:1226] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 5. Current term is 6. Ops: [4.6-5.7]
I20260807 11:10:58.033654 25926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1754) took 1383 ms. Trace:
I20260807 11:10:58.034665 25926 rpcz_store.cc:276] 0807 11:10:56.650145 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:58.031313 (+1381168us) service_pool.cc:227] Handling call
0807 11:10:58.031841 (+ 528us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:10:58.033151 (+ 1310us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:58.033545 (+ 394us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:58.022423 29779 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:58.022903 29217 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:58.024137 29644 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:58.026885 25800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 361) took 3217 ms. Trace:
W20260807 11:10:58.028913 29915 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.598s user 0.004s sys 0.023s
I20260807 11:10:57.004050 29953 raft_consensus.cc:504] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:58.032289 25931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1753) took 1385 ms. Trace:
W20260807 11:10:58.006044 25981 connection.cc:232] RPC call timeout handler was delayed by -1.4615684130694717s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19410000000000002s for a response
I20260807 11:10:58.051152 25931 rpcz_store.cc:276] 0807 11:10:56.646901 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:58.030590 (+1383689us) service_pool.cc:227] Handling call
0807 11:10:58.031129 (+ 539us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:58.031985 (+ 856us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:58.032189 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:58.020049 30029 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:58.013348 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 1.36757e+06us
W20260807 11:10:58.053126 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 51275.1us
I20260807 11:10:58.010905 25918 rpcz_store.cc:276] 0807 11:10:53.958777 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.656524 (+697747us) service_pool.cc:227] Handling call
0807 11:10:54.657037 (+ 513us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:10:54.934342 (+277305us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:54.934371 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 11:10:54.966353 (+ 31982us) log.cc:842] Serialized 52 byte log entry
0807 11:10:55.325078 (+358725us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:10:55.501300 (+176222us) log.cc:842] Serialized 14 byte log entry
0807 11:10:55.504438 (+ 3138us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:55.504553 (+ 115us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:56.714257 (+1209704us) raft_consensus.cc:1671] finished
0807 11:10:56.714284 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:56.714441 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28908,"prepare.queue_time_us":327906,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":88789,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":28767,"thread_start_us":708236,"threads_started":3,"wal-append.queue_time_us":1713485}
W20260807 11:10:58.057327 25981 connection.cc:232] RPC call timeout handler was delayed by -1.4615684130694717s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19410000000000002s for a response
I20260807 11:10:58.045354 25800 rpcz_store.cc:276] 0807 11:10:54.808888 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:55.022870 (+213982us) service_pool.cc:227] Handling call
0807 11:10:55.030243 (+ 7373us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:55.546390 (+516147us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:58.026350 (+2479960us) spinlock_profiling.cc:243] Waited 485 ms on lock 0x7b54002f7454. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:58.026668 (+ 318us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1115946240}
I20260807 11:10:58.058501 29217 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:58.059104 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39752 (request call id 1613) took 1986 ms. Trace:
I20260807 11:10:58.058710 29779 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:58.059466 29217 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:58.063076 29888 raft_consensus.cc:2726] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 11:10:58.063926 29888 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:58.064646 29893 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:58.064556 29888 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:58.065945 29978 raft_consensus.cc:482] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:58.064566 29173 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:58.063113 25872 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.291s user 0.010s sys 0.018s
I20260807 11:10:58.066725 25808 consensus_queue.cc:260] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:58.067543 25872 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.406s user 0.010s sys 0.018s
W20260807 11:10:58.067963 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 1.41742e+06us
I20260807 11:10:58.066074 29779 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:58.066803 29978 raft_consensus.cc:504] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:58.070989 25981 connection.cc:232] RPC call timeout handler was delayed by -1.4381684130057693s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1967s for a response
W20260807 11:10:58.071950 25981 connection.cc:232] RPC call timeout handler was delayed by -1.3850684128701687s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2026s for a response
W20260807 11:10:58.072853 25981 connection.cc:232] RPC call timeout handler was delayed by -1.3067684129346162s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21130000000000002s for a response
W20260807 11:10:58.076649 25981 connection.cc:232] RPC call timeout handler was delayed by -1.2905684129800647s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21309999999999998s for a response
I20260807 11:10:58.078645 25808 raft_consensus.cc:3037] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:58.063691 29217 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:10:58.083310 30023 log.cc:924] Time spent T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.070s user 0.002s sys 0.000s
I20260807 11:10:56.961512 29991 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:58.085292 26984 raft_consensus.cc:2454] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:58.085461 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 451) took 3361 ms. Trace:
W20260807 11:10:58.085857 26984 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 474) took 1434 ms (client timeout 1876 ms). Trace:
I20260807 11:10:58.085968 29991 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:58.086274 26984 rpcz_store.cc:269] 0807 11:10:56.651194 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:57.981973 (+1330779us) service_pool.cc:227] Handling call
0807 11:10:58.085785 (+103812us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:58.086611 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:58.087764 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:58.088904 29893 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:58.050315 29669 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:57.920636 26033 rpcz_store.cc:276] 0807 11:10:53.978847 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.313895 (+335048us) service_pool.cc:227] Handling call
0807 11:10:54.314258 (+ 363us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:54.345140 (+ 30882us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:54.345174 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:54.345686 (+ 512us) log.cc:842] Serialized 26 byte log entry
0807 11:10:54.712102 (+366416us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:54.712139 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.712226 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:56.411719 (+1699493us) spinlock_profiling.cc:243] Waited 1.16 ms on lock 0x7b24002c0378. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc5f4cf4 00007f3fbc5f2dfb 00007f3fbc681559 00007f3fbc6811e5 00007f3fbc68082d 00007f3fbf32f46e 00007f3fbf346c0c 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551
0807 11:10:56.980534 (+568815us) raft_consensus.cc:1671] finished
0807 11:10:56.980565 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:57.726250 (+745685us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30350,"raft.queue_time_us":31492,"raft.run_cpu_time_us":348,"raft.run_wall_time_us":33429,"spinlock_wait_cycles":2677120,"thread_start_us":366033,"threads_started":1,"wal-append.queue_time_us":475979}
I20260807 11:10:58.075965 30029 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:10:58.092384 25981 connection.cc:232] RPC call timeout handler was delayed by -1.2806684130337089s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2142s for a response
W20260807 11:10:58.071924 29876 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.228s user 0.017s sys 0.014s
I20260807 11:10:58.032714 29883 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:58.100630 25981 connection.cc:232] RPC call timeout handler was delayed by -1.2806684130337089s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2142s for a response
I20260807 11:10:58.097414 29898 leader_election.cc:288] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:58.101053 25981 connection.cc:232] RPC call timeout handler was delayed by -1.2680684130173177s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2156s for a response
I20260807 11:10:58.103960 29728 raft_consensus.cc:979] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc: : Instructing follower b22d69db45194908ad3d345eba9badd9 to start an election
I20260807 11:10:58.104041 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:58.106081 29323 raft_consensus.cc:3032] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 11:10:58.104568 25806 raft_consensus.cc:1226] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: []
W20260807 11:10:58.114482 25981 connection.cc:232] RPC call timeout handler was delayed by -1.2680684130173177s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2156s for a response
W20260807 11:10:58.120450 25981 connection.cc:232] RPC call timeout handler was delayed by -1.2401684129144996s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2187s for a response
I20260807 11:10:58.085783 26039 rpcz_store.cc:276] 0807 11:10:54.724276 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.724450 (+ 174us) service_pool.cc:227] Handling call
0807 11:10:54.724867 (+ 417us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:54.725092 (+ 225us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:54.725128 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:54.746152 (+ 21024us) log.cc:842] Serialized 26 byte log entry
0807 11:10:55.154686 (+408534us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:10:55.154738 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:55.154828 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:58.084967 (+2930139us) raft_consensus.cc:1671] finished
0807 11:10:58.085032 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:58.085286 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":361397,"threads_started":1,"wal-append.queue_time_us":986363}
I20260807 11:10:58.117903 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:58.124228 25796 raft_consensus.cc:2379] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 6: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:58.125016 30029 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:58.126775 29953 leader_election.cc:288] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:58.128969 29881 raft_consensus.cc:504] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:58.129846 29715 consensus_queue.cc:260] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:58.131171 29715 raft_consensus.cc:3037] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:58.131551 29881 leader_election.cc:288] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:58.131793 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:58.132710 26036 raft_consensus.cc:2454] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:10:58.068130 29691 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:58.114159 29323 raft_consensus.cc:729] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:10:58.135506 29691 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:58.136176 29323 consensus_queue.cc:260] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:58.069638 29915 raft_consensus.cc:686] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
W20260807 11:10:58.136742 29937 log.cc:924] Time spent T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 2.359s user 0.012s sys 0.012s
I20260807 11:10:58.146098 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 455) took 2642 ms. Trace:
I20260807 11:10:58.146235 29883 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:58.146641 26041 rpcz_store.cc:276] 0807 11:10:55.503443 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:55.618110 (+114667us) service_pool.cc:227] Handling call
0807 11:10:55.618583 (+ 473us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:55.677812 (+ 59229us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:55.677859 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:55.719188 (+ 41329us) log.cc:842] Serialized 26 byte log entry
0807 11:10:56.013437 (+294249us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:10:56.237035 (+223598us) log.cc:842] Serialized 14 byte log entry
0807 11:10:56.256033 (+ 18998us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:56.256145 (+ 112us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:58.144522 (+1888377us) raft_consensus.cc:1671] finished
0807 11:10:58.144555 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:58.145896 (+ 1341us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":42345,"prepare.queue_time_us":727388,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":52,"raft.queue_time_us":667046,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":26735,"thread_start_us":496977,"threads_started":4,"wal-append.queue_time_us":621053}
I20260807 11:10:58.124066 25920 raft_consensus.cc:2454] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
W20260807 11:10:58.151239 25981 connection.cc:232] RPC call timeout handler was delayed by -1.2401684129144996s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2187s for a response
I20260807 11:10:58.151340 26034 raft_consensus.cc:2454] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:10:58.128489 30029 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:58.163307 25981 connection.cc:232] RPC call timeout handler was delayed by -1.2356684128753841s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21920000000000003s for a response
I20260807 11:10:58.109817 29991 raft_consensus.cc:482] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:58.142021 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:58.166975 25800 raft_consensus.cc:2454] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:58.138417 29669 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:58.169739 25981 connection.cc:232] RPC call timeout handler was delayed by -1.2086684128735214s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2222s for a response
I20260807 11:10:58.137696 29323 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Advancing to term 7
W20260807 11:10:58.178588 25981 connection.cc:232] RPC call timeout handler was delayed by -1.2086684128735214s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2222s for a response
I20260807 11:10:58.139324 29589 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:58.180379 29910 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:10:54.795213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.120.132:43279 (local address 127.0.0.1:36016)
0807 11:10:55.544466 (+749253us) negotiation.cc:107] Waiting for socket to connect
0807 11:10:55.544544 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0807 11:10:55.550822 (+ 6278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:10:58.059235 (+2508413us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.23.120.132:43279: received 0 of 34 requested bytes
Metrics: {"client-negotiator.queue_time_us":635526,"thread_start_us":255990,"threads_started":1}
I20260807 11:10:58.180410 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:58.179152 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 461) took 2093 ms. Trace:
W20260807 11:10:58.183619 25981 connection.cc:232] RPC call timeout handler was delayed by -1.0115684128832072s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24409999999999998s for a response
W20260807 11:10:58.184118 25981 connection.cc:232] RPC call timeout handler was delayed by -1.0115684128832072s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24409999999999998s for a response
I20260807 11:10:58.183944 26984 raft_consensus.cc:2454] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:58.187460 26984 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 473) took 1541 ms. Trace:
I20260807 11:10:58.187788 26984 rpcz_store.cc:276] 0807 11:10:56.646531 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:58.103568 (+1457037us) service_pool.cc:227] Handling call
0807 11:10:58.187402 (+ 83834us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:10:58.136371 29644 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:58.140416 29947 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:58.141992 29173 raft_consensus.cc:1067] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 5 LEADER]: Signalling peer b22d69db45194908ad3d345eba9badd9 to start an election
W20260807 11:10:58.187815 25981 connection.cc:232] RPC call timeout handler was delayed by -1.007968412945047s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2445s for a response
I20260807 11:10:58.191273 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:58.138684 29957 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:58.185163 29883 raft_consensus.cc:482] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:58.198644 29883 raft_consensus.cc:504] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:58.202955 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:10:58.203743 25798 raft_consensus.cc:3037] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:58.203504 29991 raft_consensus.cc:504] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:58.169561 29893 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:58.202113 25800 raft_consensus.cc:3037] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:58.192499 29947 raft_consensus.cc:3037] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:58.208896 30028 raft_consensus.cc:482] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:58.209640 30028 raft_consensus.cc:504] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:58.212011 30028 leader_election.cc:288] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:58.181865 26042 rpcz_store.cc:276] 0807 11:10:56.076914 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:56.097450 (+ 20536us) service_pool.cc:227] Handling call
0807 11:10:56.123203 (+ 25753us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:56.123392 (+ 189us) raft_consensus.cc:1520] Early marking committed up to index 7
0807 11:10:56.591027 (+467635us) log.cc:842] Serialized 14 byte log entry
0807 11:10:58.070250 (+1479223us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:58.101272 (+ 31022us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:10:58.116996 (+ 15724us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:58.162688 (+ 45692us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:58.170676 (+ 7988us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":824167,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":105369,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":11114,"thread_start_us":1870932,"threads_started":3}
I20260807 11:10:58.266427 25806 raft_consensus.cc:2454] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:10:58.278241 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:58.213178 29991 leader_election.cc:288] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:10:58.287772 25750 connection.cc:232] RPC call timeout handler was delayed by -1.9828524389304221s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16690000000000002s for a response
I20260807 11:10:58.140825 29915 consensus_queue.cc:237] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:58.212983 29955 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:58.310168 25750 connection.cc:232] RPC call timeout handler was delayed by -1.8262524388264865s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18430000000000002s for a response
W20260807 11:10:58.316805 25750 connection.cc:232] RPC call timeout handler was delayed by -1.8136524388100952s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1857s for a response
I20260807 11:10:58.323375 25932 raft_consensus.cc:1226] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 2. Current term is 4. Ops: []
I20260807 11:10:58.326668 29635 raft_consensus.cc:482] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:58.327495 29635 raft_consensus.cc:504] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:58.330127 29635 leader_election.cc:288] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:58.330215 29956 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:10:55.726897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36016 (local address 127.23.120.132:43279)
0807 11:10:57.013866 (+1286969us) server_negotiation.cc:206] Beginning negotiation
0807 11:10:57.037181 (+ 23315us) server_negotiation.cc:399] Waiting for connection header
0807 11:10:57.239815 (+202634us) server_negotiation.cc:407] Connection header received
0807 11:10:57.318030 (+ 78215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:10:57.320061 (+ 2031us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:10:57.320448 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:10:57.990794 (+670346us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:10:58.149362 (+158568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36016: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":777006,"thread_start_us":97698,"threads_started":1}
I20260807 11:10:58.331610 29976 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:58.332386 29976 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:58.334057 25919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 292) took 4374 ms. Trace:
I20260807 11:10:58.334436 25919 rpcz_store.cc:276] 0807 11:10:53.959096 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.660640 (+701544us) service_pool.cc:227] Handling call
0807 11:10:54.672608 (+ 11968us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:55.610611 (+938003us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:58.321100 (+2710489us) spinlock_profiling.cc:243] Waited 936 ms on lock 0x7b540033a9d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:58.333860 (+ 12760us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2153326336}
I20260807 11:10:58.334937 29976 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:58.337967 29999 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:58.338078 29725 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:58.338904 26041 raft_consensus.cc:1226] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260807 11:10:58.338660 29725 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:58.340140 25869 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.120.132:43279 timed out after 2.146s (SENT) [suppressed 3 similar messages]
I20260807 11:10:58.340305 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:58.343762 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1741) took 3225 ms. Trace:
I20260807 11:10:58.344218 26037 rpcz_store.cc:276] 0807 11:10:55.118561 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:55.118756 (+ 195us) service_pool.cc:227] Handling call
0807 11:10:55.119186 (+ 430us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:55.119623 (+ 437us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:55.119662 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:55.145072 (+ 25410us) log.cc:842] Serialized 26 byte log entry
0807 11:10:56.616646 (+1471574us) spinlock_profiling.cc:243] Waited 8.13 ms on lock 0x7b1800407b00. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbc77e345 00007f3fc55b5a39 00007f3fbc79b8bb 00007f3fbc799cbf 00007f3fbc79be22 00007f3fbf3455bb 00007f3fbf34b063 00007f3fbf34b411 00007f3fbf6985cf
0807 11:10:57.726653 (+1110007us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:57.726773 (+ 120us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:57.727001 (+ 228us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:58.343262 (+616261us) raft_consensus.cc:1671] finished
0807 11:10:58.343290 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:58.343569 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":574,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":20497,"spinlock_wait_cycles":18694528,"thread_start_us":326120,"threads_started":1,"wal-append.queue_time_us":3135669}
I20260807 11:10:58.346570 29669 raft_consensus.cc:482] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:58.347158 29669 raft_consensus.cc:504] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:58.352052 29798 raft_consensus.cc:3032] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:10:58.352591 29798 raft_consensus.cc:729] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:58.322626 29883 leader_election.cc:288] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:58.356045 25981 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:58.359248 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:58.338615 29999 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:58.376911 25872 leader_election.cc:302] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:58.381078 29888 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:58.383210 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:58.386540 26042 raft_consensus.cc:1261] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260807 11:10:58.387548 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:10:58.392400 25750 connection.cc:232] RPC call timeout handler was delayed by -1.7623524388764054s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19140000000000001s for a response
W20260807 11:10:58.393205 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 53086.4us
I20260807 11:10:58.309002 29445 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:58.488586 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:58.488804 29445 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: NON_VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: true } }
I20260807 11:10:58.489704 26986 raft_consensus.cc:2379] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 3: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
W20260807 11:10:58.490216 26986 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 440) took 3693 ms (client timeout 2009 ms). Trace:
W20260807 11:10:58.490559 26986 rpcz_store.cc:269] 0807 11:10:54.796755 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.796912 (+ 157us) service_pool.cc:227] Handling call
0807 11:10:58.488176 (+3691264us) spinlock_profiling.cc:243] Waited 4.16 ms on lock 0x7b5800000f98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc46061b0 00007f3fc4605a94 00007f3fbdcedc04 00007f3fbdcedb92
0807 11:10:58.490086 (+ 1910us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9571584}
I20260807 11:10:58.491392 29445 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:58.492897 25803 raft_consensus.cc:2454] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:58.493291 29999 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:58.493902 29999 raft_consensus.cc:482] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:58.494252 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:58.494475 29999 raft_consensus.cc:504] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:58.495707 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:58.496382 29999 leader_election.cc:288] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:58.501034 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 159184us
I20260807 11:10:58.502502 25872 leader_election.cc:302] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:58.504290 29278 raft_consensus.cc:482] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:58.504889 29278 raft_consensus.cc:504] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:58.506108 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:58.491706 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:58.523725 29669 leader_election.cc:288] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:58.523957 29798 consensus_queue.cc:260] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:58.526046 29798 raft_consensus.cc:3037] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:58.528119 25808 raft_consensus.cc:1261] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 3. (term mismatch)
I20260807 11:10:58.528628 25808 pending_rounds.cc:76] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815: Aborting all ops after (but not including) 2
I20260807 11:10:58.528936 25808 pending_rounds.cc:97] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815: Aborting uncommitted NO_OP operation due to leader change: 6.3
I20260807 11:10:58.529161 29634 raft_consensus.cc:2781] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:58.529827 29634 raft_consensus.cc:482] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:58.530213 29634 raft_consensus.cc:3037] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
W20260807 11:10:58.533151 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 177252us
I20260807 11:10:58.450502 26042 pending_rounds.cc:76] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9: Aborting all ops after (but not including) 1
I20260807 11:10:58.535482 26042 pending_rounds.cc:97] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260807 11:10:58.535787 26042 raft_consensus.cc:2866] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260807 11:10:58.549018 25750 connection.cc:232] RPC call timeout handler was delayed by -1.4959524387959391s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.221s for a response
I20260807 11:10:58.550552 25750 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:58.551815 29851 consensus_queue.cc:1035] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 11:10:58.553532 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:58.554797 26041 raft_consensus.cc:2396] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260807 11:10:58.556201 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:58.557060 26976 raft_consensus.cc:2454] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:58.559499 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:58.560346 26037 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:58.561105 29978 leader_election.cc:288] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:58.562822 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:58.564199 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:58.542445 29991 raft_consensus.cc:2726] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:58.504890 29958 raft_consensus.cc:2781] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:10:58.570736 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:58.578962 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 44005.4us
I20260807 11:10:58.581686 25981 leader_election.cc:302] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:10:58.584947 29991 raft_consensus.cc:2726] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 11:10:58.585637 26984 raft_consensus.cc:2379] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 3: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:10:58.586705 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:58.588248 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:58.592097 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:58.592470 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:58.592662 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:58.593387 26038 raft_consensus.cc:2454] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:58.595513 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:58.596312 26035 raft_consensus.cc:2379] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 4: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:10:58.597036 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:58.597194 26041 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59"
dest_uuid: "b22d69db45194908ad3d345eba9badd9"
from {username='slave'} at 127.0.0.1:59312
I20260807 11:10:58.597625 26035 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:10:58.597972 26041 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:10:58.598469 26041 raft_consensus.cc:3037] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:58.596320 26038 raft_consensus.cc:1261] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260807 11:10:58.601465 26038 pending_rounds.cc:76] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9: Aborting all ops after (but not including) 2
I20260807 11:10:58.601776 26038 pending_rounds.cc:97] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260807 11:10:58.601728 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:58.602159 26038 raft_consensus.cc:2866] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:10:58.602581 26033 raft_consensus.cc:2454] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:58.603405 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:58.603807 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:58.607374 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 57020.2us
I20260807 11:10:58.608752 25750 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:10:58.609797 29851 raft_consensus.cc:2741] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260807 11:10:58.611867 25750 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:58.611770 29589 consensus_queue.cc:1046] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 11:10:58.612897 29589 raft_consensus.cc:3032] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 11:10:58.613330 29589 raft_consensus.cc:729] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:58.613919 25926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1755) took 1963 ms. Trace:
I20260807 11:10:58.614265 29589 consensus_queue.cc:260] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:58.614707 29947 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:58.615734 29589 raft_consensus.cc:3037] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:58.614320 25926 rpcz_store.cc:276] 0807 11:10:56.650546 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:58.069247 (+1418701us) service_pool.cc:227] Handling call
0807 11:10:58.079734 (+ 10487us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:10:58.080501 (+ 767us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:58.080531 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 11:10:58.081366 (+ 835us) log.cc:842] Serialized 52 byte log entry
0807 11:10:58.280218 (+198852us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:58.308014 (+ 27796us) log.cc:842] Serialized 14 byte log entry
0807 11:10:58.349203 (+ 41189us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:58.349287 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:58.613676 (+264389us) raft_consensus.cc:1671] finished
0807 11:10:58.613695 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:58.613834 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38630,"prepare.queue_time_us":27926,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":8498,"raft.queue_time_us":41319,"raft.run_cpu_time_us":300,"raft.run_wall_time_us":298,"thread_start_us":198034,"threads_started":1,"wal-append.queue_time_us":288794}
I20260807 11:10:58.617177 29947 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:58.593448 26042 raft_consensus.cc:2454] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:58.604319 26038 raft_consensus.cc:2454] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:10:58.529359 25808 raft_consensus.cc:2866] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260807 11:10:58.675268 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 211269us
I20260807 11:10:58.678747 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:58.584966 25981 leader_election.cc:302] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:58.678457 26035 raft_consensus.cc:2454] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 8.
I20260807 11:10:58.678433 29958 raft_consensus.cc:686] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 5 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:58.699957 25869 leader_election.cc:302] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:58.700145 29958 consensus_queue.cc:237] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:58.701208 29761 raft_consensus.cc:2726] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:10:58.703033 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:58.704363 25803 raft_consensus.cc:2379] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 6: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:10:58.704551 25869 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:10:58.706589 25798 raft_consensus.cc:2396] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260807 11:10:58.713526 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:58.714514 26036 raft_consensus.cc:2454] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
W20260807 11:10:58.674860 29857 consensus_meta.cc:287] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.598s user 0.007s sys 0.005s
I20260807 11:10:58.721200 29857 raft_consensus.cc:504] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:58.721930 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:58.722548 26042 raft_consensus.cc:3032] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:10:58.722589 26035 raft_consensus.cc:2454] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:10:58.723067 26042 raft_consensus.cc:729] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:58.723609 29857 leader_election.cc:288] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:58.724457 26042 consensus_queue.cc:260] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:58.726398 26042 raft_consensus.cc:3037] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:58.726589 25869 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:10:58.729229 30031 raft_consensus.cc:3037] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
W20260807 11:10:58.729663 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 144913us
I20260807 11:10:58.734550 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:58.735563 28964 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:58.735771 25808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1743) took 4007 ms. Trace:
I20260807 11:10:58.736599 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:58.800029 26035 raft_consensus.cc:2454] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:10:58.743769 25800 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:58.806480 26037 raft_consensus.cc:2454] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 7.
I20260807 11:10:58.803646 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:58.798972 25808 rpcz_store.cc:276] 0807 11:10:54.727898 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.730246 (+ 2348us) service_pool.cc:227] Handling call
0807 11:10:54.730580 (+ 334us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:54.923026 (+192446us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b54001dbda0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf654730 00007f3fbf6c6300 00007f3fbf6c7371 00007f3fbf6c3edf 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc
0807 11:10:58.065806 (+3142780us) spinlock_profiling.cc:243] Waited 6.82 ms on lock 0x7b54001dbda0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf65a951 00007f3fbf6bb321 00007f3fbf6bcb4e 00007f3fbf6c6c89 00007f3fbf6c7556 00007f3fbf6c3edf 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612
0807 11:10:58.735389 (+669583us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:58.735703 (+ 314us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":288888448}
I20260807 11:10:58.815171 26041 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:58.808120 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 81625us
W20260807 11:10:58.843338 26039 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.003s sys 0.019s
W20260807 11:10:58.844318 26039 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.514s user 0.003s sys 0.019s
W20260807 11:10:58.809096 26936 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.741s user 0.029s sys 0.069s
W20260807 11:10:58.845780 26936 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.777s user 0.029s sys 0.070s
W20260807 11:10:58.846146 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 1.07532e+06us
I20260807 11:10:58.849275 26041 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:58.849977 26033 raft_consensus.cc:1226] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7. Ops: []
I20260807 11:10:58.851241 29445 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:58.856948 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:58.859573 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:58.856688 29445 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:58.862661 25808 raft_consensus.cc:1226] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260807 11:10:58.869184 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:58.870064 26041 raft_consensus.cc:2396] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260807 11:10:58.870870 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:58.871668 29696 raft_consensus.cc:2726] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:10:57.976826 25917 rpcz_store.cc:276] 0807 11:10:53.843598 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.647501 (+803903us) service_pool.cc:227] Handling call
0807 11:10:54.647943 (+ 442us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:10:54.996045 (+348102us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:54.996088 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 11:10:55.001619 (+ 5531us) log.cc:842] Serialized 52 byte log entry
0807 11:10:55.338927 (+337308us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:55.339013 (+ 86us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:55.339176 (+ 163us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:56.412148 (+1072972us) raft_consensus.cc:1671] finished
0807 11:10:56.412203 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:56.653394 (+241191us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5212,"raft.queue_time_us":86412,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":5434,"thread_start_us":632091,"threads_started":2,"wal-append.queue_time_us":566101,"wal-append.run_cpu_time_us":16292,"wal-append.run_wall_time_us":1815488}
I20260807 11:10:58.876716 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:58.878018 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:10:58.878793 26938 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.695s user 0.032s sys 0.054s
W20260807 11:10:58.879223 26938 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.817s user 0.033s sys 0.054s
W20260807 11:10:58.879498 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 817021us
I20260807 11:10:58.880169 26938 leader_election.cc:302] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:10:58.881598 29930 log.cc:924] Time spent T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 2.859s user 0.024s sys 0.025s
I20260807 11:10:58.882102 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:58.882848 26035 raft_consensus.cc:3037] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:58.889685 25798 raft_consensus.cc:1226] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 7. Current term is 8. Ops: []
I20260807 11:10:58.844919 28964 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:58.891991 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1752) took 2245 ms. Trace:
I20260807 11:10:58.892428 25869 leader_election.cc:302] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:58.854729 29947 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Advancing to term 8
W20260807 11:10:58.887521 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 275733us
I20260807 11:10:58.897578 25932 rpcz_store.cc:276] 0807 11:10:56.646627 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:58.010810 (+1364183us) service_pool.cc:227] Handling call
0807 11:10:58.011500 (+ 690us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:58.323937 (+312437us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:58.888917 (+564980us) spinlock_profiling.cc:243] Waited 312 ms on lock 0x7b5400339ad4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:58.891934 (+ 3017us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":716627328}
I20260807 11:10:58.899822 25917 raft_consensus.cc:1261] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 11:10:58.905000 25803 raft_consensus.cc:1261] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260807 11:10:58.909160 26037 raft_consensus.cc:2379] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 5: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:58.912917 26041 raft_consensus.cc:2454] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
W20260807 11:10:58.908206 25918 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.813s user 0.017s sys 0.064s
W20260807 11:10:58.914099 25918 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.820s user 0.017s sys 0.064s
I20260807 11:10:58.886674 29883 raft_consensus.cc:2781] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260807 11:10:58.915433 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 67077.9us
I20260807 11:10:58.914498 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1761) took 2255 ms. Trace:
I20260807 11:10:58.906054 29977 raft_consensus.cc:2781] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:58.916002 29883 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:58.916733 29669 raft_consensus.cc:2726] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 11:10:58.987638 26035 raft_consensus.cc:2454] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:10:58.942512 29977 raft_consensus.cc:482] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:58.918160 25917 rpcz_store.cc:276] 0807 11:10:56.652053 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:58.876406 (+2224353us) service_pool.cc:227] Handling call
0807 11:10:58.876742 (+ 336us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:58.903919 (+ 27177us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:58.907882 (+ 3963us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:10:58.999809 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 112583us
I20260807 11:10:58.991549 29883 raft_consensus.cc:3037] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
W20260807 11:10:59.009919 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 95158.1us
W20260807 11:10:58.915743 29915 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.525s user 0.002s sys 0.010s
I20260807 11:10:58.994217 25806 raft_consensus.cc:2454] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 7.
I20260807 11:10:58.998929 29977 raft_consensus.cc:3037] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:59.015401 30028 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:59.033285 25981 leader_election.cc:302] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:59.041263 25750 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:59.038141 29624 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260807 11:10:59.015259 29634 raft_consensus.cc:504] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:59.045755 30045 log.cc:924] Time spent T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.080s user 0.000s sys 0.001s
I20260807 11:10:59.040416 29445 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:59.106575 26938 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:59.054620 30028 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:59.076306 29624 raft_consensus.cc:2726] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260807 11:10:59.137764 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 131132us
I20260807 11:10:59.138746 25917 raft_consensus.cc:1261] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260807 11:10:59.134116 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 57536.2us
I20260807 11:10:59.144622 25932 raft_consensus.cc:1226] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 6. Current term is 7. Ops: []
W20260807 11:10:59.156989 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 129800us
W20260807 11:10:59.193001 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 115525us
W20260807 11:10:59.166292 30063 log.cc:924] Time spent T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260807 11:10:59.166649 29692 raft_consensus.cc:2781] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:59.199332 29692 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:59.199796 29692 raft_consensus.cc:3037] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Advancing to term 6
W20260807 11:10:59.201058 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 94348.7us
W20260807 11:10:59.165277 30060 log.cc:924] Time spent T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.174s user 0.000s sys 0.000s
I20260807 11:10:59.159225 30052 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:59.224973 30052 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:59.227770 30052 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:59.220734 26042 raft_consensus.cc:1261] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260807 11:10:59.228480 26042 pending_rounds.cc:76] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9: Aborting all ops after (but not including) 2
I20260807 11:10:59.228792 26042 pending_rounds.cc:97] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260807 11:10:59.229095 26042 raft_consensus.cc:2866] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:10:59.219457 29624 consensus_queue.cc:1035] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:10:59.224427 25869 leader_election.cc:302] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:59.147728 30031 raft_consensus.cc:2726] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260807 11:10:59.232558 25926 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.040s sys 0.060s
W20260807 11:10:59.232999 25926 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.547s user 0.040s sys 0.060s
I20260807 11:10:59.193457 29991 raft_consensus.cc:482] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
W20260807 11:10:59.246300 30066 log.cc:924] Time spent T dacbdb7bffca4346a1e00e7b1e90fdea P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20260807 11:10:59.248626 29893 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:59.250854 25797 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1762) took 3405 ms (client timeout 2280 ms). Trace:
W20260807 11:10:59.251260 25797 rpcz_store.cc:269] 0807 11:10:55.845101 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:55.845769 (+ 668us) service_pool.cc:227] Handling call
0807 11:10:59.250563 (+3404794us) spinlock_profiling.cc:243] Waited 2.11 s on lock 0x7b54000483d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:59.250764 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4844624128}
I20260807 11:10:59.252986 26980 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 465) took 5407 ms. Trace:
I20260807 11:10:59.253384 26980 rpcz_store.cc:276] 0807 11:10:53.845258 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.098938 (+253680us) service_pool.cc:227] Handling call
0807 11:10:54.111564 (+ 12626us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:55.356183 (+1244619us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:59.252524 (+3896341us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54004718d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:59.252776 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2846718208}
I20260807 11:10:59.255030 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:59.260524 30028 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:59.262118 29776 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:10:59.263389 25981 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:59.265839 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 249842us
I20260807 11:10:59.267237 25750 leader_election.cc:302] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:59.267884 29991 raft_consensus.cc:504] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:59.269076 29725 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:59.271186 25752 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:10:59.272203 25920 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1760) took 2620 ms (client timeout 2033 ms). Trace:
W20260807 11:10:59.272627 25920 rpcz_store.cc:269] 0807 11:10:56.651798 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:56.706404 (+ 54606us) service_pool.cc:227] Handling call
0807 11:10:59.272016 (+2565612us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b5400185e54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:59.272122 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":282589568}
I20260807 11:10:59.273030 25916 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 305) took 3178 ms. Trace:
I20260807 11:10:59.273383 25916 rpcz_store.cc:276] 0807 11:10:56.094779 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:56.296267 (+201488us) service_pool.cc:227] Handling call
0807 11:10:56.390902 (+ 94635us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:57.953355 (+1562453us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:59.272777 (+1319422us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54001def54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:10:59.272974 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2963767552}
I20260807 11:10:59.273898 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:59.274617 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:59.275270 25916 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:10:59.277374 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:10:59.275270 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:59.280933 30031 raft_consensus.cc:2726] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 11:10:59.272225 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:59.255786 26980 raft_consensus.cc:2454] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 7.
I20260807 11:10:59.284047 26033 raft_consensus.cc:2454] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 8.
W20260807 11:10:59.292948 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 47887.6us
I20260807 11:10:59.293814 25752 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:59.290287 25872 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:10:59.298453 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 92857.5us
I20260807 11:10:59.298928 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1757) took 1021 ms. Trace:
I20260807 11:10:59.301853 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:59.304030 25981 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:59.322818 25750 leader_election.cc:302] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:10:59.325271 25919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1758) took 2674 ms. Trace:
W20260807 11:10:59.359002 26979 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 472) took 2954 ms (client timeout 1892 ms). Trace:
I20260807 11:10:59.327104 29634 leader_election.cc:288] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:59.365206 25920 raft_consensus.cc:1226] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260807 11:10:59.366353 25872 leader_election.cc:302] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:59.367702 25872 leader_election.cc:302] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:59.368544 25872 leader_election.cc:302] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:59.323346 29966 raft_consensus.cc:2781] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:59.374377 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:10:59.361622 26979 rpcz_store.cc:269] 0807 11:10:56.301937 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:56.315597 (+ 13660us) service_pool.cc:227] Handling call
0807 11:10:59.256463 (+2940866us) spinlock_profiling.cc:243] Waited 246 ms on lock 0x7b540072f454. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:59.256637 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":566668032}
W20260807 11:10:59.374092 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 87051.5us
I20260807 11:10:59.350739 29857 raft_consensus.cc:2781] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Leader election won for term 8
W20260807 11:10:59.389139 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 85446us
I20260807 11:10:59.391819 25869 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:59.299427 26039 rpcz_store.cc:276] 0807 11:10:58.277903 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:58.279233 (+ 1330us) service_pool.cc:227] Handling call
0807 11:10:58.303818 (+ 24585us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:58.329499 (+ 25681us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 11:10:58.848491 (+518992us) log.cc:842] Serialized 14 byte log entry
0807 11:10:59.071670 (+223179us) log.cc:842] Serialized 14 byte log entry
0807 11:10:59.298463 (+226793us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:59.298530 (+ 67us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:10:59.298599 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:59.298684 (+ 85us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:59.298823 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9895,"prepare.queue_time_us":658585,"prepare.run_cpu_time_us":123,"prepare.run_wall_time_us":18728,"raft.queue_time_us":174964,"raft.run_cpu_time_us":365,"raft.run_wall_time_us":9150,"spinlock_wait_cycles":43140992,"thread_start_us":890737,"threads_started":3,"wal-append.queue_time_us":360429}
I20260807 11:10:59.388350 25752 leader_election.cc:302] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:59.361337 25919 rpcz_store.cc:276] 0807 11:10:56.651289 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:58.335496 (+1684207us) service_pool.cc:227] Handling call
0807 11:10:58.335919 (+ 423us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:10:58.535293 (+199374us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:58.535338 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 11:10:58.621822 (+ 86484us) log.cc:842] Serialized 52 byte log entry
0807 11:10:59.000686 (+378864us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:59.296750 (+296064us) log.cc:842] Serialized 14 byte log entry
0807 11:10:59.324822 (+ 28072us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:59.324926 (+ 104us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:59.324982 (+ 56us) raft_consensus.cc:1671] finished
0807 11:10:59.324998 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:59.325198 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":301417,"raft.queue_time_us":263879,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":95,"thread_start_us":645101,"threads_started":2,"wal-append.queue_time_us":441578}
I20260807 11:10:59.360155 30027 raft_consensus.cc:2726] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 11:10:59.404932 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:59.405756 30030 raft_consensus.cc:2781] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 11:10:59.406286 30030 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:59.406677 30030 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:10:59.408857 26042 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:10:59.410101 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:59.410769 26042 raft_consensus.cc:2454] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:10:59.411832 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:59.412707 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:10:59.376235 29972 raft_consensus.cc:2781] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:10:59.376459 29634 raft_consensus.cc:2781] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:59.376085 29966 raft_consensus.cc:482] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 11:10:59.415930 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 54891.6us
I20260807 11:10:59.406584 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:59.413916 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:10:59.393848 29893 raft_consensus.cc:2781] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260807 11:10:59.449164 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 151613us
I20260807 11:10:59.449455 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:59.358841 29609 consensus_queue.cc:1035] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.120s
I20260807 11:10:59.419691 29972 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:59.419490 29634 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:59.455281 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:59.412609 26042 raft_consensus.cc:2454] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:10:59.457158 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:59.457336 25752 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:59.458541 30085 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
W20260807 11:10:59.451463 26936 connection.cc:232] RPC call timeout handler was delayed by -1.752716425806284s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16060000000000002s for a response
I20260807 11:10:59.465195 26039 raft_consensus.cc:2396] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
W20260807 11:10:59.465515 26936 connection.cc:232] RPC call timeout handler was delayed by -1.6519164259079844s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1718s for a response
I20260807 11:10:59.442905 29966 raft_consensus.cc:3037] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:59.517043 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:59.591460 25981 leader_election.cc:302] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:10:59.596832 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 133339us
I20260807 11:10:59.599399 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:59.454437 29893 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:59.454448 29857 raft_consensus.cc:686] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 8 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:59.517174 29972 raft_consensus.cc:3037] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:59.604002 25920 raft_consensus.cc:3037] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:59.610937 25916 raft_consensus.cc:1226] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: []
I20260807 11:10:59.605813 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:59.596410 29883 consensus_meta.cc:287] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.570s user 0.004s sys 0.008s
I20260807 11:10:59.443495 30031 raft_consensus.cc:2781] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election won for term 5
W20260807 11:10:59.523785 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 66507.5us
I20260807 11:10:59.620493 25752 leader_election.cc:302] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:59.623512 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:59.633999 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:59.463251 29680 raft_consensus.cc:2781] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:59.462909 29624 raft_consensus.cc:2781] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:59.611943 29893 raft_consensus.cc:3037] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:59.517395 29634 raft_consensus.cc:3037] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
W20260807 11:10:59.640012 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 271510us
I20260807 11:10:59.652858 25916 raft_consensus.cc:2379] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 6: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:10:59.655571 26039 raft_consensus.cc:1226] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:59.655311 25872 leader_election.cc:302] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:59.607734 30081 raft_consensus.cc:2781] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:59.632243 29696 raft_consensus.cc:2781] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:10:59.657904 25872 leader_election.cc:302] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:10:59.662428 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:59.637147 30090 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:59.658934 26936 leader_election.cc:302] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:10:59.649075 29680 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:59.651850 29624 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:59.696786 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:10:59.700361 30075 log.cc:924] Time spent T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.243s user 0.000s sys 0.000s
I20260807 11:10:59.662510 29696 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:59.701915 29696 raft_consensus.cc:3037] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:59.702157 25981 leader_election.cc:302] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:59.661942 30081 raft_consensus.cc:482] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:59.705134 30081 raft_consensus.cc:3037] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:59.705971 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:10:59.706842 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 49013.2us
I20260807 11:10:59.707260 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:59.707695 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1849) took 1110 ms. Trace:
I20260807 11:10:59.707808 25919 raft_consensus.cc:3037] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:10:59.708051 26038 rpcz_store.cc:276] 0807 11:10:58.668797 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:58.682360 (+ 13563us) service_pool.cc:227] Handling call
0807 11:10:58.831075 (+148715us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:58.885366 (+ 54291us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:58.885412 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:58.885966 (+ 554us) log.cc:842] Serialized 26 byte log entry
0807 11:10:59.232013 (+346047us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:59.232067 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:59.232157 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:59.707484 (+475327us) raft_consensus.cc:1671] finished
0807 11:10:59.707505 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:59.707641 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":112,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":156,"thread_start_us":345549,"threads_started":1,"wal-append.queue_time_us":440796}
I20260807 11:10:59.709226 29813 raft_consensus.cc:2781] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader election won for term 7
I20260807 11:10:59.709908 29813 raft_consensus.cc:686] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 7 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:10:59.625098 29883 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:59.695369 29680 raft_consensus.cc:3037] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:59.718492 25872 leader_election.cc:302] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:10:59.692468 30070 log.cc:924] Time spent T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.248s user 0.001s sys 0.000s
I20260807 11:10:59.606884 30085 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:59.654304 29669 raft_consensus.cc:2726] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 11:10:59.662760 29609 raft_consensus.cc:2781] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:10:59.656337 29636 raft_consensus.cc:2781] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260807 11:10:59.733312 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 269977us
I20260807 11:10:59.728431 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:59.732039 25920 raft_consensus.cc:2454] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
W20260807 11:10:59.736088 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 98500.5us
I20260807 11:10:59.712647 29624 raft_consensus.cc:3037] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:59.661847 29991 leader_election.cc:288] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:10:59.742143 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:59.742303 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:10:59.742457 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:59.743675 25869 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:59.744170 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1742) took 5694 ms. Trace:
I20260807 11:10:59.740322 30048 raft_consensus.cc:2781] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:10:59.746711 30048 raft_consensus.cc:482] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:59.747066 30048 raft_consensus.cc:3037] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:10:59.733736 29609 raft_consensus.cc:482] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:10:59.720324 29725 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:10:59.733485 29636 raft_consensus.cc:482] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
W20260807 11:10:59.792559 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 74144us
I20260807 11:10:59.799793 25872 leader_election.cc:302] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:10:59.799194 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:10:59.794735 26039 raft_consensus.cc:1226] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: []
I20260807 11:10:59.797694 25920 raft_consensus.cc:3037] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
W20260807 11:10:59.790911 30070 log.cc:924] Time spent T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260807 11:10:59.744668 25921 rpcz_store.cc:276] 0807 11:10:53.894954 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.644676 (+749722us) service_pool.cc:227] Handling call
0807 11:10:55.028033 (+383357us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:55.032539 (+ 4506us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:59.415647 (+4383108us) spinlock_profiling.cc:243] Waited 382 ms on lock 0x7b5400052b50. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:59.588855 (+173208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":879263616}
W20260807 11:10:59.806187 29776 consensus_meta.cc:287] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.544s user 0.005s sys 0.007s
I20260807 11:10:59.806648 29883 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:59.807753 29883 raft_consensus.cc:2781] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:59.808105 29883 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:10:59.809487 26975 raft_consensus.cc:2396] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260807 11:10:59.810027 29760 raft_consensus.cc:482] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b22d69db45194908ad3d345eba9badd9)
I20260807 11:10:59.810557 29760 raft_consensus.cc:3037] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:59.802698 29813 consensus_queue.cc:237] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:59.799299 29991 raft_consensus.cc:2781] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260807 11:10:59.814781 29977 consensus_meta.cc:287] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.793s user 0.011s sys 0.000s
I20260807 11:10:59.815845 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:10:59.816565 25916 raft_consensus.cc:3037] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:59.817431 25981 leader_election.cc:302] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:59.808513 29883 raft_consensus.cc:3037] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:59.795697 29609 raft_consensus.cc:3037] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:10:59.845359 25932 raft_consensus.cc:1226] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:59.801409 29636 raft_consensus.cc:3037] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:10:59.801859 29798 raft_consensus.cc:2781] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:10:59.853163 29798 raft_consensus.cc:686] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:10:59.854383 29798 consensus_queue.cc:237] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:10:59.860137 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 205525us
I20260807 11:10:59.814918 29991 raft_consensus.cc:482] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:10:59.864231 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:10:59.870287 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 126685us
I20260807 11:10:59.872380 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:59.814558 29654 leader_election.cc:288] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:10:59.875064 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 84381.2us
I20260807 11:10:59.739135 30090 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:59.862812 29991 raft_consensus.cc:3037] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
W20260807 11:10:59.882619 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 82898.3us
I20260807 11:10:59.863723 30031 raft_consensus.cc:686] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 5 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:10:59.819887 29977 raft_consensus.cc:504] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:59.888837 29732 consensus_queue.cc:1035] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:10:59.798777 29857 consensus_queue.cc:237] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:59.867589 29278 leader_election.cc:288] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:59.892489 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1857) took 1154 ms. Trace:
I20260807 11:10:59.838058 29725 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:59.898811 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:10:59.902384 25920 raft_consensus.cc:2454] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
W20260807 11:10:59.908624 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 119322us
I20260807 11:10:59.908855 26039 raft_consensus.cc:1261] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 9 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
W20260807 11:10:59.907279 26035 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.002s sys 0.007s
I20260807 11:10:59.889546 30030 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:10:59.895144 29915 raft_consensus.cc:2726] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260807 11:10:59.917747 26035 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.532s user 0.003s sys 0.007s
I20260807 11:10:59.933303 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:10:59.899546 30109 raft_consensus.cc:2781] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:10:59.959162 26986 raft_consensus.cc:1226] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 2. Current term is 3. Ops: []
I20260807 11:10:59.944630 25981 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:10:59.906327 30105 raft_consensus.cc:2781] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:10:59.988447 25919 raft_consensus.cc:2454] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
W20260807 11:10:59.969832 25796 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1575) took 4195 ms (client timeout 1517 ms). Trace:
W20260807 11:10:59.925374 30071 log.cc:924] Time spent T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.415s user 0.000s sys 0.001s
W20260807 11:11:00.012220 25796 rpcz_store.cc:269] 0807 11:10:55.774194 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:55.774810 (+ 616us) service_pool.cc:227] Handling call
0807 11:10:59.969571 (+4194761us) spinlock_profiling.cc:243] Waited 2.25 s on lock 0x7b54002f5154. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:59.969724 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5169414272}
I20260807 11:11:00.016172 29278 raft_consensus.cc:482] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
W20260807 11:10:59.957794 26974 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 437) took 5186 ms (client timeout 1847 ms). Trace:
I20260807 11:10:59.947789 29696 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:00.016973 29278 raft_consensus.cc:504] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:11:00.017199 26974 rpcz_store.cc:269] 0807 11:10:54.770808 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.771965 (+ 1157us) service_pool.cc:227] Handling call
0807 11:10:59.957335 (+5185370us) spinlock_profiling.cc:243] Waited 3.26 s on lock 0x7b540074f9d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:10:59.957556 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7486975872}
I20260807 11:10:59.952965 30108 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:00.019025 29278 leader_election.cc:288] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:00.018903 30108 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:10:59.963941 30069 log.cc:924] Time spent T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.699s user 0.002s sys 0.001s
W20260807 11:10:59.954479 29947 consensus_meta.cc:287] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 1.046s user 0.010s sys 0.002s
I20260807 11:11:00.031687 29947 raft_consensus.cc:2726] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260807 11:10:59.962366 29878 log.cc:924] Time spent T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 4.739s user 0.049s sys 0.089s
I20260807 11:11:00.023389 29851 raft_consensus.cc:482] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:00.033061 29851 raft_consensus.cc:504] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:00.033865 29725 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:00.034554 29636 raft_consensus.cc:504] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:00.035224 29851 leader_election.cc:288] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:10:58.059571 25925 rpcz_store.cc:276] 0807 11:10:56.074647 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:56.088488 (+ 13841us) service_pool.cc:227] Handling call
0807 11:10:56.122856 (+ 34368us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:56.123089 (+ 233us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:56.123115 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:56.123537 (+ 422us) log.cc:842] Serialized 26 byte log entry
0807 11:10:56.532219 (+408682us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:56.896127 (+363908us) log.cc:842] Serialized 14 byte log entry
0807 11:10:58.013785 (+1117658us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:58.035157 (+ 21372us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:58.052831 (+ 17674us) raft_consensus.cc:1671] finished
0807 11:10:58.052851 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:58.058884 (+ 6033us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":808621,"prepare.queue_time_us":1523472,"raft.queue_time_us":1080469,"raft.run_cpu_time_us":358,"raft.run_wall_time_us":44159,"thread_start_us":679794,"threads_started":3,"wal-append.queue_time_us":703684}
W20260807 11:11:00.023329 25750 connection.cc:232] RPC call timeout handler was delayed by -1.4667823871131986s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1897s for a response
I20260807 11:10:59.948347 29732 consensus_queue.cc:1046] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.076s
I20260807 11:11:00.024724 29977 leader_election.cc:288] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:10:59.917956 30113 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:10:59.968887 30085 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:00.026889 26979 raft_consensus.cc:1226] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 LEADER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260807 11:11:00.029560 30108 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:10:59.974472 29908 log.cc:924] Time spent T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 4.346s user 0.031s sys 0.022s
I20260807 11:10:59.929098 29917 consensus_queue.cc:1035] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.104s
W20260807 11:11:00.025425 26985 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 436) took 5302 ms (client timeout 3776 ms). Trace:
I20260807 11:10:59.934242 29634 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:59.925271 30031 consensus_queue.cc:237] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:10:59.920013 29417 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:11:00.036641 25750 connection.cc:232] RPC call timeout handler was delayed by -1.3443823871202767s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2033s for a response
I20260807 11:11:00.188342 29624 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:00.116124 29692 consensus_meta.cc:287] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.883s user 0.009s sys 0.001s
I20260807 11:10:59.956213 29972 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:00.154798 26987 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 468) took 6285 ms. Trace:
I20260807 11:11:00.180122 25916 raft_consensus.cc:2454] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
W20260807 11:11:00.094851 30080 log.cc:924] Time spent T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.357s user 0.000s sys 0.002s
I20260807 11:11:00.037029 25931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1756) took 3386 ms. Trace:
I20260807 11:11:00.187937 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:11:00.260875 30081 consensus_meta.cc:287] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.555s user 0.005s sys 0.005s
I20260807 11:11:00.360121 29636 leader_election.cc:288] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:00.010795 30105 raft_consensus.cc:482] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 11:11:00.349364 26985 rpcz_store.cc:269] 0807 11:10:54.770189 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.770360 (+ 171us) service_pool.cc:227] Handling call
0807 11:11:00.024921 (+5254561us) spinlock_profiling.cc:243] Waited 552 ms on lock 0x7b54006b03d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:00.025176 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1269463424}
I20260807 11:11:00.096446 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 308) took 1033 ms. Trace:
W20260807 11:11:00.359488 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 337242us
I20260807 11:11:00.117362 25926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1759) took 3465 ms. Trace:
W20260807 11:11:00.401772 25752 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.003s sys 0.050s
W20260807 11:11:00.412075 25752 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.537s user 0.003s sys 0.050s
W20260807 11:11:00.412501 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 549190us
I20260807 11:11:00.353004 30113 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:00.094663 25801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53324 (request call id 1569) took 6223 ms. Trace:
I20260807 11:10:59.966929 30090 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:00.409855 25926 rpcz_store.cc:276] 0807 11:10:56.651535 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:58.616497 (+1964962us) service_pool.cc:227] Handling call
0807 11:10:58.616982 (+ 485us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:58.684278 (+ 67296us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:58.684311 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:58.685215 (+ 904us) log.cc:842] Serialized 26 byte log entry
0807 11:10:59.233478 (+548263us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:59.233511 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:59.233569 (+ 58us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:00.117130 (+883561us) raft_consensus.cc:1671] finished
0807 11:11:00.117149 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:00.117300 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":61062,"raft.queue_time_us":66461,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":700,"spinlock_wait_cycles":1307520,"thread_start_us":547718,"threads_started":1,"wal-append.queue_time_us":674023}
W20260807 11:11:00.400470 29958 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s user 0.004s sys 0.023s
I20260807 11:11:00.110255 30030 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:00.190797 30048 raft_consensus.cc:504] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:00.401454 25931 rpcz_store.cc:276] 0807 11:10:56.650817 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:58.075903 (+1425086us) service_pool.cc:227] Handling call
0807 11:10:58.078567 (+ 2664us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:58.133497 (+ 54930us) raft_consensus.cc:1520] Early marking committed up to index 8
0807 11:10:58.133937 (+ 440us) log.cc:842] Serialized 14 byte log entry
0807 11:10:58.337321 (+203384us) log.cc:842] Serialized 14 byte log entry
0807 11:10:58.337483 (+ 162us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:58.337524 (+ 41us) raft_consensus.cc:1631] Marking committed up to 8
0807 11:10:58.337570 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:00.036808 (+1699238us) spinlock_profiling.cc:243] Waited 43.4 ms on lock 0x7b54003624d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:00.036852 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:00.036975 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26713,"prepare.queue_time_us":5043,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":99,"raft.queue_time_us":28678,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":182,"spinlock_wait_cycles":99900288,"thread_start_us":175894,"threads_started":1,"wal-append.queue_time_us":354087,"wal-append.run_cpu_time_us":2127,"wal-append.run_wall_time_us":1030642}
I20260807 11:11:00.408501 25917 rpcz_store.cc:276] 0807 11:10:59.063477 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:59.222928 (+159451us) service_pool.cc:227] Handling call
0807 11:10:59.237521 (+ 14593us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:10:59.291718 (+ 54197us) raft_consensus.cc:1520] Early marking committed up to index 6
0807 11:10:59.644226 (+352508us) log.cc:842] Serialized 14 byte log entry
0807 11:10:59.860072 (+215846us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 11:10:59.862042 (+ 1970us) log.cc:842] Serialized 52 byte log entry
0807 11:10:59.869690 (+ 7648us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:10:59.869744 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:59.869836 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:00.096217 (+226381us) raft_consensus.cc:1671] finished
0807 11:11:00.096249 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:00.096388 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30863,"prepare.queue_time_us":413259,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"raft.queue_time_us":93976,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":2892,"spinlock_wait_cycles":657792,"thread_start_us":437197,"threads_started":2,"wal-append.queue_time_us":270731}
I20260807 11:11:00.114037 29696 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:00.453841 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:00.407367 29939 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:00.455147 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:00.465811 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:11:00.429574 29958 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.615s user 0.005s sys 0.023s
I20260807 11:11:00.466621 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:00.293350 29883 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:00.472574 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:10:59.949301 29760 raft_consensus.cc:504] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:00.419061 25801 rpcz_store.cc:276] 0807 11:10:53.871066 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.459092 (+588026us) service_pool.cc:227] Handling call
0807 11:10:54.533905 (+ 74813us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:54.541883 (+ 7978us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:54.541914 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:54.565127 (+ 23213us) log.cc:842] Serialized 26 byte log entry
0807 11:10:54.977856 (+412729us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:10:54.977909 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:54.978005 (+ 96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:59.961360 (+4983355us) spinlock_profiling.cc:243] Waited 5.08 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:11:00.037002 (+ 75642us) raft_consensus.cc:1671] finished
0807 11:11:00.037020 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:00.094416 (+ 57396us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":85583,"raft.queue_time_us":7763,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":1490,"spinlock_wait_cycles":11687808,"thread_start_us":377529,"threads_started":1,"wal-append.queue_time_us":545363}
I20260807 11:11:00.464135 26936 leader_election.cc:302] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:00.415000 30105 raft_consensus.cc:3037] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:00.401639 26987 rpcz_store.cc:276] 0807 11:10:53.869629 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.132129 (+262500us) service_pool.cc:227] Handling call
0807 11:10:54.138177 (+ 6048us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:10:54.949222 (+811045us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:54.949251 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 11:10:54.998000 (+ 48749us) log.cc:842] Serialized 52 byte log entry
0807 11:10:55.337587 (+339587us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:10:55.337628 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:00.124407 (+4786779us) spinlock_profiling.cc:243] Waited 440 ms on lock 0x7b54006b44d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:00.124475 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:00.124540 (+ 65us) raft_consensus.cc:1671] finished
0807 11:11:00.124551 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:00.154588 (+ 30037us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":89874,"raft.queue_time_us":149961,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":10705,"spinlock_wait_cycles":1010859264,"thread_start_us":543662,"threads_started":2,"wal-append.queue_time_us":578023,"wal-append.run_cpu_time_us":16465,"wal-append.run_wall_time_us":1766206}
I20260807 11:11:00.331323 29991 raft_consensus.cc:504] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:00.466159 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:00.099874 29893 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:00.481787 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:00.483268 25799 raft_consensus.cc:2396] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260807 11:11:00.100673 29680 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:00.344303 29609 raft_consensus.cc:504] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:00.476342 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:00.401393 29692 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:00.496898 26985 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:00.493079 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:00.109483 29966 raft_consensus.cc:504] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:00.498569 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:00.499060 29634 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:00.499205 25799 raft_consensus.cc:1226] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 4. Current term is 5. Ops: []
I20260807 11:11:00.501255 29609 leader_election.cc:288] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:00.502945 29644 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:00.508359 30113 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:00.508885 26033 raft_consensus.cc:3037] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:00.509068 25803 raft_consensus.cc:1261] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260807 11:11:00.509414 29940 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:00.414654 30081 raft_consensus.cc:504] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:00.512926 29267 raft_consensus.cc:3037] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Advancing to term 6
W20260807 11:11:00.515581 30116 log.cc:924] Time spent T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.563s user 0.000s sys 0.012s
W20260807 11:11:00.498539 25870 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.005s sys 0.021s
W20260807 11:11:00.533133 25870 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.631s user 0.006s sys 0.021s
I20260807 11:11:00.407459 29417 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:00.523195 25801 raft_consensus.cc:2454] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:11:00.528152 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:00.535431 25800 raft_consensus.cc:2379] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 5: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:11:00.528959 26980 raft_consensus.cc:1226] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 5. Current term is 6. Ops: []
I20260807 11:11:00.476873 29696 consensus_queue.cc:1046] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260807 11:11:00.521850 29802 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:00.525552 29654 raft_consensus.cc:3037] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:11:00.544190 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:11:00.544853 29765 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 4.923s user 0.055s sys 0.061s
I20260807 11:11:00.545202 25796 raft_consensus.cc:2396] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 9 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260807 11:11:00.545584 29906 raft_consensus.cc:482] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:00.546147 29906 raft_consensus.cc:504] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:00.546871 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:00.547668 25797 raft_consensus.cc:2379] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 3: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
W20260807 11:11:00.552958 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 128213us
I20260807 11:11:00.499457 29939 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:11:00.542032 29798 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.009s sys 0.048s
W20260807 11:11:00.545218 29765 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 5.051s user 0.055s sys 0.061s
I20260807 11:11:00.529618 29947 consensus_queue.cc:1046] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260807 11:11:00.544682 30048 leader_election.cc:288] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:11:00.578594 26938 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.019s sys 0.014s
W20260807 11:11:00.579649 26938 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.554s user 0.019s sys 0.014s
I20260807 11:11:00.579886 26986 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 471) took 4274 ms. Trace:
W20260807 11:11:00.557605 29857 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.027s sys 0.040s
W20260807 11:11:00.580982 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 572267us
W20260807 11:11:00.571785 29798 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.646s user 0.009s sys 0.049s
I20260807 11:11:00.600620 26986 rpcz_store.cc:276] 0807 11:10:56.301064 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:58.554795 (+2253731us) service_pool.cc:227] Handling call
0807 11:10:59.958918 (+1404123us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:59.977746 (+ 18828us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:00.575371 (+597625us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b540074f9d0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:00.575517 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3228167424}
W20260807 11:11:00.572885 29813 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.006s sys 0.014s
I20260807 11:11:00.608018 26033 raft_consensus.cc:2454] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:11:00.585754 25797 pending_rounds.cc:76] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815: Aborting all ops after (but not including) 2
I20260807 11:11:00.612259 25797 pending_rounds.cc:97] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815: Aborting uncommitted NO_OP operation due to leader change: 5.3
W20260807 11:11:00.574994 29776 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.684s user 0.005s sys 0.016s
I20260807 11:11:00.613031 25797 raft_consensus.cc:2866] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:11:00.537212 29940 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:00.636997 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:00.636643 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:00.643316 25796 raft_consensus.cc:3037] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:00.585537 29972 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:00.611500 29696 consensus_queue.cc:1046] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
W20260807 11:11:00.611840 29813 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.610s user 0.006s sys 0.015s
I20260807 11:11:00.644861 26033 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:11:00.609351 29624 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:11:00.640645 29776 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.747s user 0.008s sys 0.016s
I20260807 11:11:00.568993 29802 raft_consensus.cc:504] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:00.610710 29765 consensus_queue.cc:1225] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Peer 4fd6381544574ddca559ce63b9243efc log is divergent from this leader: its last log entry 4.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260807 11:11:00.677626 25798 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
W20260807 11:11:00.601238 29857 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.626s user 0.030s sys 0.043s
I20260807 11:11:00.637205 29906 leader_election.cc:288] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:00.682137 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:00.682367 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:00.678635 26938 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:00.646123 29798 raft_consensus.cc:482] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:00.701460 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:00.655443 29760 leader_election.cc:288] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:00.707142 30130 raft_consensus.cc:3032] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:11:00.701176 29917 raft_consensus.cc:2781] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 11:11:00.655143 30090 consensus_queue.cc:1035] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.052s
I20260807 11:11:00.708485 29798 raft_consensus.cc:504] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:00.677085 29883 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:00.682318 29893 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:00.711210 29940 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:00.706283 29680 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:00.712917 26984 raft_consensus.cc:2454] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:11:00.681195 29991 leader_election.cc:288] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:00.714376 26936 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:00.650182 30048 consensus_queue.cc:1046] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260807 11:11:00.714385 29917 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:00.720376 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:00.700280 30081 leader_election.cc:288] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:00.743523 25803 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:00.703665 29692 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:00.725179 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:00.745335 25801 raft_consensus.cc:2379] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 6: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:11:00.716269 30130 raft_consensus.cc:729] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:00.715364 29966 leader_election.cc:288] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:11:00.722469 26974 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.702s user 0.011s sys 0.048s
I20260807 11:11:00.758252 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:00.760849 30048 raft_consensus.cc:3032] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 11:11:00.779284 25801 raft_consensus.cc:2379] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 6: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:11:00.779510 30048 raft_consensus.cc:729] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:00.772854 29267 raft_consensus.cc:2781] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:11:00.796901 30048 consensus_queue.cc:260] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:00.798394 30048 raft_consensus.cc:3037] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:00.723737 29323 raft_consensus.cc:482] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:00.773784 29939 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:00.774307 29798 leader_election.cc:288] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:00.775745 29802 leader_election.cc:288] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:00.775961 28964 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:00.783644 26985 raft_consensus.cc:2454] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 8.
W20260807 11:11:00.780210 26974 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.760s user 0.011s sys 0.053s
W20260807 11:11:00.795696 26976 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 470) took 4496 ms (client timeout 2783 ms). Trace:
I20260807 11:11:00.795876 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:00.746565 29917 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Advancing to term 9
I20260807 11:11:00.801770 25803 raft_consensus.cc:2454] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 8.
I20260807 11:11:00.801983 29267 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:00.824440 25796 raft_consensus.cc:2454] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 7.
I20260807 11:11:00.823360 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:00.826277 25803 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:11:00.715627 29977 consensus_queue.cc:1046] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
W20260807 11:11:00.802922 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 371913us
I20260807 11:11:00.817167 29893 raft_consensus.cc:482] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:00.840483 25796 raft_consensus.cc:1261] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 11:11:00.844167 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:00.832840 29267 raft_consensus.cc:3037] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:00.850327 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:00.824005 29875 consensus_queue.cc:1046] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260807 11:11:00.852216 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:00.852726 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:00.853263 25931 raft_consensus.cc:1261] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260807 11:11:00.809780 29323 raft_consensus.cc:504] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:00.837731 28964 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:11:00.817411 26976 rpcz_store.cc:269] 0807 11:10:56.298862 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:56.315734 (+ 16872us) service_pool.cc:227] Handling call
0807 11:11:00.795427 (+4479693us) spinlock_profiling.cc:243] Waited 1.64 s on lock 0x7b54002c1354. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:00.795596 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3766267136}
I20260807 11:11:00.875612 25796 raft_consensus.cc:2379] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 6: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:11:00.876021 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:00.878036 25920 raft_consensus.cc:1226] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 5. Current term is 6. Ops: []
W20260807 11:11:00.876400 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 332999us
W20260807 11:11:00.881664 26046 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.023s sys 0.023s
W20260807 11:11:00.882014 26046 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.688s user 0.024s sys 0.023s
I20260807 11:11:00.843518 29893 raft_consensus.cc:504] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:00.883579 25803 raft_consensus.cc:2454] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:11:00.908541 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:00.909184 25803 raft_consensus.cc:3037] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:11:00.910674 25869 leader_election.cc:302] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:00.904361 28964 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:00.919652 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:00.920298 28964 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:00.925472 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1865) took 2014 ms. Trace:
I20260807 11:11:00.925953 26042 rpcz_store.cc:276] 0807 11:10:58.911571 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:59.625334 (+713763us) service_pool.cc:227] Handling call
0807 11:10:59.647605 (+ 22271us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:59.670148 (+ 22543us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 11:10:59.740830 (+ 70682us) log.cc:842] Serialized 14 byte log entry
0807 11:11:00.925026 (+1184196us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:00.925101 (+ 75us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:00.925172 (+ 71us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:00.925256 (+ 84us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:00.925404 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1043981,"prepare.queue_time_us":103994,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":14642,"thread_start_us":76906,"threads_started":1,"wal-append.queue_time_us":129264}
I20260807 11:11:00.930112 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1867) took 1919 ms. Trace:
W20260807 11:11:00.930747 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 1.00493e+06us
I20260807 11:11:00.936357 25869 leader_election.cc:302] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:00.928750 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 473) took 1657 ms. Trace:
I20260807 11:11:00.939213 26042 rpcz_store.cc:276] 0807 11:10:59.270902 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:00.927793 (+1656891us) service_pool.cc:227] Handling call
0807 11:11:00.928183 (+ 390us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:00.928387 (+ 204us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:00.928421 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:00.928475 (+ 54us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:00.928516 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:00.928576 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:00.928687 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:00.920405 25800 raft_consensus.cc:3037] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:00.884964 30038 raft_consensus.cc:2726] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:11:00.930465 26038 rpcz_store.cc:276] 0807 11:10:59.011107 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:59.709560 (+698453us) service_pool.cc:227] Handling call
0807 11:10:59.709894 (+ 334us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:59.710158 (+ 264us) raft_consensus.cc:1520] Early marking committed up to index 7
0807 11:10:59.794139 (+ 83981us) log.cc:842] Serialized 14 byte log entry
0807 11:11:00.929730 (+1135591us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:00.929778 (+ 48us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:00.929875 (+ 97us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:00.929947 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:00.930050 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1063313,"prepare.queue_time_us":89443,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":21930,"raft.queue_time_us":1033264,"raft.run_cpu_time_us":50,"raft.run_wall_time_us":49,"thread_start_us":50527,"threads_started":1,"wal-append.queue_time_us":164153}
I20260807 11:11:00.943742 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:00.954103 25869 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:00.955668 25869 leader_election.cc:302] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:00.958568 25872 leader_election.cc:302] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:00.945797 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:11:00.933728 29950 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.739s user 0.029s sys 0.050s
I20260807 11:11:00.885607 30105 raft_consensus.cc:504] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:00.995704 25869 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:00.943212 29883 raft_consensus.cc:2726] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:11:00.943590 29680 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:11:01.016970 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 46965us
W20260807 11:11:00.974447 30020 log.cc:924] Time spent T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 2.872s user 0.017s sys 0.006s
W20260807 11:11:00.967901 29950 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.905s user 0.029s sys 0.050s
W20260807 11:11:01.040356 26034 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43246 (request call id 463) took 3152 ms (client timeout 2152 ms). Trace:
W20260807 11:11:01.042069 26034 rpcz_store.cc:269] 0807 11:10:57.862932 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:57.864421 (+ 1489us) service_pool.cc:227] Handling call
0807 11:11:01.015481 (+3151060us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b540038e854. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:01.015605 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":463595776}
I20260807 11:11:00.973316 30030 raft_consensus.cc:2726] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 11:11:00.973855 29813 raft_consensus.cc:2726] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:11:01.007187 30112 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:01.009191 30102 raft_consensus.cc:482] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:01.024881 29991 raft_consensus.cc:2726] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260807 11:11:01.095729 26041 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59312 (request call id 1783) took 1733 ms (client timeout 1675 ms). Trace:
W20260807 11:11:01.096318 26041 rpcz_store.cc:269] 0807 11:10:59.362685 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:59.404055 (+ 41370us) service_pool.cc:227] Handling call
0807 11:11:01.095427 (+1691372us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b540029de54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:01.095599 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3155612416}
I20260807 11:11:01.097746 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:01.098569 26041 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:11:01.041898 29680 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:01.100839 29680 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:11:01.101578 30141 log.cc:924] Time spent T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260807 11:11:01.101694 29972 raft_consensus.cc:3032] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 11:11:01.102363 29972 raft_consensus.cc:729] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:01.021142 30081 consensus_queue.cc:1046] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.082s
I20260807 11:11:00.875252 30130 consensus_queue.cc:260] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:01.003917 30129 raft_consensus.cc:2726] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:11:01.105657 25806 raft_consensus.cc:1226] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 4. Current term is 5. Ops: []
I20260807 11:11:01.106119 29950 raft_consensus.cc:3032] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 11:11:01.106518 29950 raft_consensus.cc:729] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:11:01.107390 29950 consensus_queue.cc:260] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:01.109251 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:01.109793 26034 raft_consensus.cc:3037] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:11:01.091910 29608 raft_consensus.cc:2781] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:11:01.112488 29608 raft_consensus.cc:686] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:01.020931 30113 raft_consensus.cc:2726] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:01.113457 29608 consensus_queue.cc:237] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:01.092926 30137 log.cc:924] Time spent T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.212s user 0.000s sys 0.001s
I20260807 11:11:01.094884 30105 leader_election.cc:288] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:01.095525 30102 raft_consensus.cc:504] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:01.116097 26036 raft_consensus.cc:1203] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Deduplicated request from leader. Original: 3.5->[5.6-5.6] Dedup: 5.6->[]
I20260807 11:11:01.121934 26036 raft_consensus.cc:1226] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260807 11:11:01.122112 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 464) took 3258 ms. Trace:
W20260807 11:11:01.094429 26984 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 453) took 2535 ms (client timeout 2520 ms). Trace:
I20260807 11:11:01.133153 25800 raft_consensus.cc:2454] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
W20260807 11:11:01.133431 26984 rpcz_store.cc:269] 0807 11:10:58.558759 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:58.587915 (+ 29156us) service_pool.cc:227] Handling call
0807 11:11:01.094192 (+2506277us) spinlock_profiling.cc:243] Waited 2.12 s on lock 0x7b54007409d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:01.094342 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4870285696}
I20260807 11:11:01.134807 25800 raft_consensus.cc:1226] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 5. Current term is 6. Ops: []
I20260807 11:11:01.122870 25803 raft_consensus.cc:2454] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:11:01.008859 29644 consensus_queue.cc:1035] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.083s
W20260807 11:11:01.137784 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 206354us
I20260807 11:11:01.139569 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:01.140403 26041 raft_consensus.cc:3037] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:01.083173 29947 consensus_queue.cc:1035] T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
I20260807 11:11:01.132529 26046 rpcz_store.cc:276] 0807 11:10:57.863576 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:57.864958 (+ 1382us) service_pool.cc:227] Handling call
0807 11:10:57.865705 (+ 747us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:10:57.952642 (+ 86937us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:57.952702 (+ 60us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 11:11:00.192717 (+2240015us) spinlock_profiling.cc:243] Waited 31.5 ms on lock 0x7b2c00107d68. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbe0a7e11 00007f3fbe0b7442 00007f3fbf6ffc11 00007f3fbf6c458a 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772
0807 11:11:00.193391 (+ 674us) log.cc:842] Serialized 52 byte log entry
0807 11:11:00.882440 (+689049us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:11:00.882490 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:00.882597 (+ 107us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:01.115777 (+233180us) raft_consensus.cc:1671] finished
0807 11:11:01.115796 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:01.122031 (+ 6235us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15301,"raft.queue_time_us":168117,"raft.run_cpu_time_us":383,"raft.run_wall_time_us":36110,"spinlock_wait_cycles":72485120,"thread_start_us":769287,"threads_started":2,"wal-append.queue_time_us":691845}
I20260807 11:11:01.164438 25869 leader_election.cc:302] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:01.143685 30085 raft_consensus.cc:482] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:01.179979 25869 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:01.181912 25869 leader_election.cc:302] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:01.189261 29875 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:11:01.165644 26975 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 442) took 5763 ms (client timeout 2146 ms). Trace:
I20260807 11:11:01.116238 30112 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:01.200459 26034 raft_consensus.cc:2454] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
W20260807 11:11:01.162309 29654 consensus_meta.cc:287] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.609s user 0.006s sys 0.006s
I20260807 11:11:01.204456 25869 leader_election.cc:302] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:01.181944 26974 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50942 (request call id 459) took 1589 ms. Trace:
I20260807 11:11:01.190395 29875 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:01.219547 29847 consensus_queue.cc:1035] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260807 11:11:01.220444 30172 raft_consensus.cc:3032] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 11:11:01.239820 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:01.241278 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:01.244875 30113 raft_consensus.cc:2726] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:11:01.254369 26936 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:01.254813 29875 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:11:01.255956 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 51599.2us
I20260807 11:11:01.183226 30094 raft_consensus.cc:2726] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:11:01.263064 25869 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:11:01.220957 30129 raft_consensus.cc:2726] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:11:01.268747 26936 leader_election.cc:302] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:11:01.267336 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 48016.5us
I20260807 11:11:01.282994 26041 raft_consensus.cc:2454] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 7.
I20260807 11:11:01.272074 29915 consensus_queue.cc:1035] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20260807 11:11:01.242547 30109 raft_consensus.cc:686] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:11:01.288698 25869 leader_election.cc:302] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:01.290040 25869 leader_election.cc:302] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:01.289669 30109 consensus_queue.cc:237] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:01.291746 25869 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:11:01.292841 30126 raft_consensus.cc:2781] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:11:01.293291 30126 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:01.293643 30126 raft_consensus.cc:3037] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:01.293861 29847 raft_consensus.cc:2726] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 11:11:01.294545 29847 raft_consensus.cc:2781] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Leader election won for term 8
I20260807 11:11:01.295022 29847 raft_consensus.cc:686] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 8 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:01.298034 30108 raft_consensus.cc:3032] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 11:11:01.298552 30108 raft_consensus.cc:729] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:01.296028 29847 consensus_queue.cc:237] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:01.299489 30108 consensus_queue.cc:260] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:01.303336 30126 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:01.304169 29267 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:01.245031 30172 raft_consensus.cc:729] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:01.305681 30126 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:01.310608 30172 consensus_queue.cc:260] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:01.242228 26974 rpcz_store.cc:276] 0807 11:10:59.587844 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:00.018437 (+430593us) service_pool.cc:227] Handling call
0807 11:11:00.018908 (+ 471us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:00.019228 (+ 320us) raft_consensus.cc:1520] Early marking committed up to index 6
0807 11:11:00.019658 (+ 430us) log.cc:842] Serialized 14 byte log entry
0807 11:11:01.137921 (+1118263us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:01.140385 (+ 2464us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:01.145586 (+ 5201us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:01.151760 (+ 6174us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:01.176805 (+ 25045us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":2750,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":369024,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":225,"thread_start_us":1103471,"threads_started":2,"wal-append.queue_time_us":884766}
I20260807 11:11:01.272243 29813 raft_consensus.cc:2726] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:01.312413 30172 raft_consensus.cc:3037] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:01.250735 29654 raft_consensus.cc:2726] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:11:01.317472 29417 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:01.286938 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:11:01.322225 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 53590.4us
I20260807 11:11:01.327687 26034 raft_consensus.cc:3037] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:01.322650 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:01.293704 29883 raft_consensus.cc:2726] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 11:11:01.329035 26041 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:11:01.332139 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 471) took 2061 ms. Trace:
I20260807 11:11:01.317329 29267 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:01.273005 30113 consensus_queue.cc:1046] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20260807 11:11:01.285494 29893 leader_election.cc:288] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:01.334522 26033 rpcz_store.cc:276] 0807 11:10:59.270292 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:00.695896 (+1425604us) service_pool.cc:227] Handling call
0807 11:11:00.699010 (+ 3114us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:00.705036 (+ 6026us) raft_consensus.cc:1520] Early marking committed up to index 7
0807 11:11:01.019101 (+314065us) log.cc:842] Serialized 14 byte log entry
0807 11:11:01.331707 (+312606us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:01.331774 (+ 67us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:01.331844 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:01.331930 (+ 86us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:01.332071 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":432462,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":15726,"raft.queue_time_us":640,"thread_start_us":579561,"threads_started":2}
W20260807 11:11:01.342105 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 181991us
I20260807 11:11:01.341624 26936 leader_election.cc:302] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:01.240211 29972 consensus_queue.cc:260] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:01.329432 30162 raft_consensus.cc:2726] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:01.221684 30085 raft_consensus.cc:504] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:01.351292 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:01.350521 25918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1757) took 4699 ms. Trace:
I20260807 11:11:01.336292 30173 raft_consensus.cc:2781] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:11:01.363315 26041 raft_consensus.cc:2396] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260807 11:11:01.363641 26035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1860) took 2622 ms. Trace:
I20260807 11:11:01.364430 25918 rpcz_store.cc:276] 0807 11:10:56.651059 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:58.086917 (+1435858us) service_pool.cc:227] Handling call
0807 11:10:58.087426 (+ 509us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:58.087946 (+ 520us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 11:10:58.088546 (+ 600us) log.cc:842] Serialized 14 byte log entry
0807 11:10:58.931794 (+843248us) log.cc:842] Serialized 14 byte log entry
0807 11:10:58.934600 (+ 2806us) log.cc:842] Serialized 14 byte log entry
0807 11:10:59.971906 (+1037306us) spinlock_profiling.cc:243] Waited 35.5 ms on lock 0x7b1800318100. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbc79c764 00007f3fbc79fb11 00007f3fbc79fac9 00007f3fbc79fa59 00007f3fbc79fa21 00007f3fbc79ed1d 00007f3fc5a00204 00007f3fc5a00039 00007f3fbc77edc6
0807 11:11:00.028631 (+ 56725us) spinlock_profiling.cc:243] Waited 31.8 ms on lock 0x7b1800318100. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbc79c638 00007f3fbc79fb11 00007f3fbc79fac9 00007f3fbc79fa59 00007f3fbc79fa21 00007f3fbc79ed1d 00007f3fc5a00204 00007f3fc5a00039 00007f3fbc77edc6
0807 11:11:00.933193 (+904562us) spinlock_profiling.cc:243] Waited 30.2 ms on lock 0x7b1800318100. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbc77e345 00007f3fc55b5a39 00007f3fbc79b8bb 00007f3fbc799cbf 00007f3fbc799155 00007f3fbf6bc3d0 00007f3fbf6ce576 00007f3fbf6df0d7 00007f3fbf6df058
0807 11:11:01.349247 (+416054us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:01.349328 (+ 81us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:01.350203 (+ 875us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:01.350313 (+ 110us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:01.350450 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1040001,"prepare.run_cpu_time_us":138,"prepare.run_wall_time_us":140,"raft.queue_time_us":30903,"raft.run_cpu_time_us":383,"raft.run_wall_time_us":26087,"spinlock_wait_cycles":224235520,"thread_start_us":1236182,"threads_started":2,"wal-append.queue_time_us":987756,"wal-append.run_cpu_time_us":2369,"wal-append.run_wall_time_us":1178568}
W20260807 11:11:01.211182 26975 rpcz_store.cc:269] 0807 11:10:55.401761 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:55.412613 (+ 10852us) service_pool.cc:227] Handling call
0807 11:11:01.158210 (+5745597us) spinlock_profiling.cc:243] Waited 4.26 s on lock 0x7b54006b26d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:01.165541 (+ 7331us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9799014784}
I20260807 11:11:01.352238 29730 raft_consensus.cc:2726] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:01.382843 26046 raft_consensus.cc:3037] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:01.377324 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:01.377111 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:01.384727 26041 raft_consensus.cc:2396] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 8 because replica has last-logged OpId of term: 7 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260807 11:11:01.362040 30173 raft_consensus.cc:482] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:01.366585 30160 raft_consensus.cc:2781] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:11:01.373855 30102 leader_election.cc:288] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:11:01.409401 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 117705us
I20260807 11:11:01.423739 25869 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:11:01.419266 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:01.374341 29883 raft_consensus.cc:3032] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:11:01.414841 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:01.346266 30161 consensus_queue.cc:1035] T f31f65eddcdf48f6a64e0a1ddc0c6695 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20260807 11:11:01.422804 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:01.366412 26035 rpcz_store.cc:276] 0807 11:10:58.738681 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:58.995895 (+257214us) service_pool.cc:227] Handling call
0807 11:10:59.009052 (+ 13157us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:59.037501 (+ 28449us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 11:10:59.318525 (+281024us) log.cc:842] Serialized 14 byte log entry
0807 11:10:59.934190 (+615665us) log.cc:842] Serialized 14 byte log entry
0807 11:11:01.108444 (+1174254us) spinlock_profiling.cc:243] Waited 9.51 ms on lock 0x7b1800408a00. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbc79c764 00007f3fbc79fb11 00007f3fbc79fac9 00007f3fbc79fa59 00007f3fbc79fa21 00007f3fbc79ed1d 00007f3fc5a00204 00007f3fc5a00039 00007f3fbc77edc6
0807 11:11:01.152267 (+ 43823us) log.cc:842] Serialized 14 byte log entry
0807 11:11:01.360643 (+208376us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:01.360694 (+ 51us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:01.360743 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:01.360832 (+ 89us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:01.361074 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":995829,"prepare.queue_time_us":338617,"prepare.run_cpu_time_us":248,"prepare.run_wall_time_us":29843,"raft.queue_time_us":1000526,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":26390,"spinlock_wait_cycles":21868800,"thread_start_us":1241764,"threads_started":4,"wal-append.queue_time_us":562411,"wal-append.run_cpu_time_us":1989,"wal-append.run_wall_time_us":1032862}
I20260807 11:11:01.426693 26033 raft_consensus.cc:2454] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:11:01.438690 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:01.430190 25918 raft_consensus.cc:3037] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:01.389209 29972 raft_consensus.cc:3037] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:01.441825 26041 raft_consensus.cc:3037] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:01.385762 29654 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:01.444047 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1780) took 2082 ms. Trace:
I20260807 11:11:01.447463 26038 rpcz_store.cc:276] 0807 11:10:59.362132 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:00.950700 (+1588568us) service_pool.cc:227] Handling call
0807 11:11:01.014077 (+ 63377us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:01.023857 (+ 9780us) raft_consensus.cc:1520] Early marking committed up to index 2
0807 11:11:01.324250 (+300393us) log.cc:842] Serialized 14 byte log entry
0807 11:11:01.443658 (+119408us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:01.443725 (+ 67us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:11:01.443785 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:01.443874 (+ 89us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:01.443993 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13821,"prepare.queue_time_us":274993,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":8630,"raft.queue_time_us":16709,"thread_start_us":320348,"threads_started":2}
I20260807 11:11:01.446410 25925 raft_consensus.cc:2454] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 8.
I20260807 11:11:01.415872 30173 raft_consensus.cc:3037] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:11:01.452260 26936 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:01.461575 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:01.442652 30113 raft_consensus.cc:2781] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader election won for term 7
I20260807 11:11:01.462371 26035 raft_consensus.cc:2454] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:11:01.465921 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1774) took 2107 ms. Trace:
I20260807 11:11:01.421526 30162 consensus_queue.cc:1035] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.053s
I20260807 11:11:01.448269 30160 raft_consensus.cc:686] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
W20260807 11:11:01.465489 30048 consensus_meta.cc:287] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.667s user 0.014s sys 0.000s
I20260807 11:11:01.477304 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:01.440605 29883 raft_consensus.cc:729] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:01.464365 29267 raft_consensus.cc:2781] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 11:11:01.553231 26035 raft_consensus.cc:2454] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:11:01.486112 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:01.538022 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:01.482429 26034 raft_consensus.cc:2454] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:11:01.555311 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:11:01.477946 29917 consensus_meta.cc:287] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.645s user 0.011s sys 0.000s
I20260807 11:11:01.554797 26033 raft_consensus.cc:2454] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:11:01.590724 25869 leader_election.cc:302] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:01.562247 26041 raft_consensus.cc:2454] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:11:01.594491 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:01.595626 26035 raft_consensus.cc:2379] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 6: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:11:01.492169 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:01.494982 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:01.602259 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:11:01.597251 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 87918.7us
I20260807 11:11:01.492672 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:01.541613 30113 raft_consensus.cc:686] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [term 7 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:01.601320 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:11:01.619855 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 67637.6us
I20260807 11:11:01.600479 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:01.620993 26041 raft_consensus.cc:3037] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:01.560212 30178 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:01.461895 29654 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:01.623114 26038 raft_consensus.cc:2396] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260807 11:11:01.621415 26938 leader_election.cc:302] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:11:01.623183 26035 raft_consensus.cc:2454] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:11:01.463539 30182 consensus_queue.cc:1046] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.171s
I20260807 11:11:01.627139 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:01.552531 29267 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:01.621028 26975 raft_consensus.cc:3037] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:01.636289 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:01.636610 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:01.636945 26034 raft_consensus.cc:3037] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:11:01.637315 26033 raft_consensus.cc:3037] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:01.560024 30112 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:01.591815 29917 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:01.624702 29847 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:01.647733 26046 raft_consensus.cc:2454] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:11:01.645645 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:01.650300 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:01.650913 26038 raft_consensus.cc:3037] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 6
I20260807 11:11:01.630039 29970 raft_consensus.cc:2726] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 11:11:01.648974 26035 raft_consensus.cc:2379] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 6: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:11:01.632853 29267 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Advancing to term 9
I20260807 11:11:01.653074 29847 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:01.656994 30113 consensus_queue.cc:237] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:01.660324 30174 log.cc:924] Time spent T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.185s user 0.001s sys 0.000s
I20260807 11:11:01.661909 25981 leader_election.cc:302] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:01.663625 25981 leader_election.cc:302] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:01.663818 29654 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:11:01.665079 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 74474.9us
I20260807 11:11:01.669273 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:01.669657 25872 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:11:01.669948 25919 raft_consensus.cc:2454] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 7.
I20260807 11:11:01.671208 26936 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:01.675494 30130 raft_consensus.cc:3037] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:11:01.626737 30178 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:01.678216 30105 raft_consensus.cc:2781] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:01.678341 30178 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:01.678711 30105 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:01.679219 30105 raft_consensus.cc:3037] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:01.680855 30160 consensus_queue.cc:237] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:01.626260 29883 consensus_queue.cc:260] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:01.684094 29883 raft_consensus.cc:3037] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:11:01.687377 29950 raft_consensus.cc:3037] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:01.687310 30173 raft_consensus.cc:504] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:01.689260 30162 raft_consensus.cc:2781] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 11:11:01.689718 30162 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:01.690150 30162 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:01.628090 29991 consensus_queue.cc:1035] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.068s
I20260807 11:11:01.695801 30109 raft_consensus.cc:2726] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:11:01.696497 30109 raft_consensus.cc:2726] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 11:11:01.651930 30048 raft_consensus.cc:3032] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 11:11:01.697247 30048 raft_consensus.cc:729] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:01.698091 30048 consensus_queue.cc:260] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:01.699360 30048 raft_consensus.cc:3037] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:10:59.896797 26037 rpcz_store.cc:276] 0807 11:10:58.737662 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:58.914363 (+176701us) service_pool.cc:227] Handling call
0807 11:10:58.989661 (+ 75298us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:59.040313 (+ 50652us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 11:10:59.414256 (+373943us) log.cc:842] Serialized 14 byte log entry
0807 11:10:59.892059 (+477803us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:10:59.892126 (+ 67us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:10:59.892194 (+ 68us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:59.892276 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:10:59.892418 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6783,"prepare.queue_time_us":91981,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":61,"raft.queue_time_us":91100,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":26662,"spinlock_wait_cycles":25425920,"thread_start_us":760665,"threads_started":3,"wal-append.queue_time_us":411319}
I20260807 11:11:01.700237 25918 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:11:01.655370 30182 raft_consensus.cc:2781] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:11:01.700711 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:01.722661 26037 raft_consensus.cc:3037] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
W20260807 11:11:01.712998 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 49467.3us
I20260807 11:11:01.725445 25981 leader_election.cc:302] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:11:01.727034 25806 raft_consensus.cc:1261] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260807 11:11:01.727454 25806 pending_rounds.cc:76] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815: Aborting all ops after (but not including) 6
I20260807 11:11:01.727659 25806 pending_rounds.cc:97] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260807 11:11:01.727882 25806 raft_consensus.cc:2866] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:11:01.648576 30085 leader_election.cc:288] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:01.730979 26041 raft_consensus.cc:2454] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:11:01.738272 26035 raft_consensus.cc:1226] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 4. Current term is 5. Ops: []
I20260807 11:11:01.739168 26035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1781) took 2376 ms. Trace:
I20260807 11:11:01.739578 26035 rpcz_store.cc:276] 0807 11:10:59.362376 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:01.709799 (+2347423us) service_pool.cc:227] Handling call
0807 11:11:01.734233 (+ 24434us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:01.738856 (+ 4623us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:01.739102 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:01.742175 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:01.724257 30182 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:01.748677 29267 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:01.749374 30182 raft_consensus.cc:3037] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:01.749403 30085 raft_consensus.cc:2741] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260807 11:11:01.750561 29917 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 9 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:01.750892 29267 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 9 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:01.759011 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:01.761622 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:01.762599 25918 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Advancing to term 9
I20260807 11:11:01.761360 26033 raft_consensus.cc:2454] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 7.
I20260807 11:11:01.768886 25872 leader_election.cc:302] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:01.770646 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:01.772704 26038 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
W20260807 11:11:01.769843 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 44526.9us
I20260807 11:11:01.773949 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:01.775019 25981 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:01.775624 25872 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:11:01.777401 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 156082us
I20260807 11:11:01.778246 26938 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:11:01.780505 26938 leader_election.cc:302] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:11:01.782744 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:01.785465 26033 raft_consensus.cc:2454] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:11:01.786309 26938 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:01.786646 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:01.782358 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:01.793444 26035 raft_consensus.cc:3037] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:01.792596 26037 raft_consensus.cc:2454] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:11:01.783735 29970 raft_consensus.cc:2726] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:11:01.796207 30171 raft_consensus.cc:2781] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:11:01.785290 30112 raft_consensus.cc:2726] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 11:11:01.796705 30171 raft_consensus.cc:482] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:01.797608 25872 leader_election.cc:302] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:01.798007 30171 raft_consensus.cc:3037] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:11:01.798653 25872 leader_election.cc:302] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:01.799482 29893 raft_consensus.cc:2781] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:01.799942 30178 raft_consensus.cc:2781] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:11:01.800489 30178 raft_consensus.cc:686] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:01.801440 30178 consensus_queue.cc:237] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:01.811685 26033 raft_consensus.cc:2379] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 7: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:11:01.813305 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:01.814400 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:01.777300 29915 raft_consensus.cc:2781] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader election won for term 7
I20260807 11:11:01.817978 26041 raft_consensus.cc:3037] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 6
I20260807 11:11:01.819159 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:01.815608 26034 raft_consensus.cc:2454] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:11:01.819700 26038 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:11:01.820864 25872 leader_election.cc:302] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:11:01.823385 26034 raft_consensus.cc:1261] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 11:11:01.824102 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 474) took 2357 ms. Trace:
I20260807 11:11:01.824478 26034 rpcz_store.cc:276] 0807 11:10:59.466391 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:01.822646 (+2356255us) service_pool.cc:227] Handling call
0807 11:11:01.822999 (+ 353us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:01.823832 (+ 833us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:01.824044 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:01.814083 29917 raft_consensus.cc:2781] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:11:01.799943 29893 raft_consensus.cc:482] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:01.829953 25918 raft_consensus.cc:2454] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 9.
I20260807 11:11:01.786849 30172 raft_consensus.cc:2781] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:11:01.833228 26034 raft_consensus.cc:1261] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 11:11:01.833479 26936 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:01.834311 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:01.834803 26038 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Advancing to term 9
I20260807 11:11:01.837378 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 476) took 2223 ms. Trace:
I20260807 11:11:01.837827 26033 rpcz_store.cc:276] 0807 11:10:59.614370 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:01.826713 (+2212343us) service_pool.cc:227] Handling call
0807 11:11:01.827070 (+ 357us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:01.837023 (+ 9953us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:01.837056 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:01.837097 (+ 41us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:01.837134 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:01.837191 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:01.837303 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:01.830062 29970 raft_consensus.cc:2781] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election won for term 5
W20260807 11:11:01.827786 30090 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.806s user 0.013s sys 0.008s
I20260807 11:11:01.850741 26041 raft_consensus.cc:2454] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:11:01.853303 25981 leader_election.cc:302] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:01.853699 25872 leader_election.cc:302] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:01.851312 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 475) took 2231 ms. Trace:
I20260807 11:11:01.854980 26034 rpcz_store.cc:276] 0807 11:10:59.602555 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:01.825394 (+2222839us) service_pool.cc:227] Handling call
0807 11:11:01.832887 (+ 7493us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:01.833678 (+ 791us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:01.833882 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:01.839584 29970 raft_consensus.cc:686] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 5 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:01.839609 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 477) took 2204 ms. Trace:
I20260807 11:11:01.860580 26033 rpcz_store.cc:276] 0807 11:10:59.635556 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:01.838828 (+2203272us) service_pool.cc:227] Handling call
0807 11:11:01.839108 (+ 280us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:01.839291 (+ 183us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:01.839315 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:01.839353 (+ 38us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:01.839389 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:01.839455 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:01.839558 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:01.866284 25797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1800) took 1314 ms. Trace:
I20260807 11:11:01.863325 26035 raft_consensus.cc:2454] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:11:01.870489 25872 leader_election.cc:302] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:01.872345 26041 raft_consensus.cc:1226] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 5. Current term is 6. Ops: []
W20260807 11:11:01.852312 30090 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.835s user 0.013s sys 0.009s
I20260807 11:11:01.833936 29893 raft_consensus.cc:3037] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:01.877153 26038 raft_consensus.cc:2454] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 9.
I20260807 11:11:01.832515 29915 raft_consensus.cc:686] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 7 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
W20260807 11:11:01.886379 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 52608.1us
W20260807 11:11:01.901230 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 768075us
I20260807 11:11:01.866801 25797 rpcz_store.cc:276] 0807 11:11:00.552299 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:00.560116 (+ 7817us) service_pool.cc:227] Handling call
0807 11:11:00.575633 (+ 15517us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:00.651102 (+ 75469us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:00.651132 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:00.689683 (+ 38551us) log.cc:842] Serialized 26 byte log entry
0807 11:11:01.073007 (+383324us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:01.081166 (+ 8159us) log.cc:842] Serialized 14 byte log entry
0807 11:11:01.840605 (+759439us) spinlock_profiling.cc:243] Waited 4.96 ms on lock 0x7b18000157c0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbc7134ca 00007f3fc5df4f99 00007f3fbc5d7990 00007f3fbc799864 00007f3fbc799155 00007f3fbf6bc3d0 00007f3fbf6ce576 00007f3fbf6df0d7 00007f3fbf6df058
0807 11:11:01.856721 (+ 16116us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:01.858532 (+ 1811us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:01.858598 (+ 66us) raft_consensus.cc:1671] finished
0807 11:11:01.866010 (+ 7412us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:01.866196 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":515,"prepare.queue_time_us":8535,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":82,"raft.queue_time_us":775397,"spinlock_wait_cycles":11404032,"thread_start_us":308695,"threads_started":1,"wal-append.queue_time_us":504346}
I20260807 11:11:01.860419 30129 raft_consensus.cc:2781] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:11:01.852730 26974 raft_consensus.cc:1261] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260807 11:11:01.901494 29917 raft_consensus.cc:686] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:01.909107 25981 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:11:01.862368 29991 raft_consensus.cc:2726] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:11:01.881695 30094 raft_consensus.cc:2781] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:11:01.919960 26038 raft_consensus.cc:1261] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260807 11:11:01.916523 25981 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:11:01.917572 25869 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:11:01.921018 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 80200us
I20260807 11:11:01.920518 30169 raft_consensus.cc:2781] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:01.904095 30160 consensus_queue.cc:1035] T d47975fbb6e3412aaf146ed0794f986c P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260807 11:11:01.939329 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 10 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:01.938750 26938 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:01.940160 26038 raft_consensus.cc:2396] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 10 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260807 11:11:01.924367 29939 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:01.942883 25981 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:01.943791 30112 raft_consensus.cc:2726] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:11:01.932929 29847 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:01.945034 26975 raft_consensus.cc:2454] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 7.
I20260807 11:11:01.945143 25750 leader_election.cc:302] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:01.945351 25752 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:01.945515 25981 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:01.946250 25750 leader_election.cc:302] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:11:01.946355 25752 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:01.946719 30112 raft_consensus.cc:2726] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 11:11:01.947408 25750 leader_election.cc:302] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:01.948199 25981 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:11:01.949301 30112 raft_consensus.cc:2741] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20260807 11:11:01.953611 25750 leader_election.cc:302] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:01.960034 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:01.935165 30172 raft_consensus.cc:686] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:01.968789 30172 consensus_queue.cc:237] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:01.970377 30113 raft_consensus.cc:2781] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:11:01.971259 25803 raft_consensus.cc:2396] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260807 11:11:01.935458 29991 raft_consensus.cc:2741] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260807 11:11:01.929654 30171 raft_consensus.cc:504] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:11:01.972366 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 138989us
I20260807 11:11:01.974380 25981 leader_election.cc:302] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:01.938046 30169 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:01.977250 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:01.978036 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:01.972937 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:01.983969 25796 raft_consensus.cc:2379] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 7: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:11:01.985122 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:01.985499 25797 raft_consensus.cc:2379] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 6: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
W20260807 11:11:01.982560 29977 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.129s user 0.020s sys 0.010s
I20260807 11:11:01.952107 30216 raft_consensus.cc:2781] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 FOLLOWER]: Leader election won for term 9
I20260807 11:11:01.973572 29760 raft_consensus.cc:2726] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:01.985731 25803 raft_consensus.cc:2362] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5. Re-sending same reply.
I20260807 11:11:01.992563 30129 raft_consensus.cc:686] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
W20260807 11:11:01.990487 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 73004.2us
I20260807 11:11:01.975283 29765 raft_consensus.cc:2781] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:11:01.990901 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:01.981964 29798 raft_consensus.cc:2781] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:11:02.003258 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:02.003813 25797 raft_consensus.cc:3037] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:01.984642 29732 raft_consensus.cc:2726] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260807 11:11:02.010310 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 64002.5us
I20260807 11:11:01.982074 30113 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
W20260807 11:11:02.006861 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 54985us
I20260807 11:11:01.984748 28964 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:02.011940 28964 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:02.015143 28964 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:01.985210 30169 raft_consensus.cc:3037] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:01.987892 29680 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:02.016141 29680 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:02.018889 29680 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:01.984601 30094 raft_consensus.cc:686] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:02.025419 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:01.942283 29970 consensus_queue.cc:237] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:01.991465 30218 raft_consensus.cc:2781] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 FOLLOWER]: Leader election won for term 9
I20260807 11:11:02.046195 25798 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:11:02.078805 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:02.080614 25798 raft_consensus.cc:2454] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 8.
I20260807 11:11:01.992247 29940 consensus_queue.cc:1046] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.058s
I20260807 11:11:01.957894 29915 consensus_queue.cc:237] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:01.972707 29939 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:02.001888 30112 raft_consensus.cc:2726] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:11:02.048122 25750 leader_election.cc:302] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:11:02.068936 25796 raft_consensus.cc:3037] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:02.066367 25806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 376) took 1247 ms. Trace:
W20260807 11:11:02.079284 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 52311.2us
I20260807 11:11:01.988106 30090 consensus_queue.cc:1035] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.039s
W20260807 11:11:01.994259 29977 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.145s user 0.020s sys 0.010s
I20260807 11:11:02.106212 25806 rpcz_store.cc:276] 0807 11:11:00.818824 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:01.106618 (+287794us) service_pool.cc:227] Handling call
0807 11:11:01.116170 (+ 9552us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:01.734806 (+618636us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:02.066125 (+331319us) spinlock_profiling.cc:243] Waited 610 ms on lock 0x7b5400266e54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:02.066309 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1402657664}
I20260807 11:11:02.095436 25869 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:02.098613 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:02.010035 29972 raft_consensus.cc:2726] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:11:02.098524 25752 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:11:02.054701 30213 log.cc:924] Time spent T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.133s user 0.000s sys 0.000s
I20260807 11:11:02.025583 30113 raft_consensus.cc:3037] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:11:02.113483 25798 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Advancing to term 9
I20260807 11:11:02.113081 30219 consensus_queue.cc:1035] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 11:11:02.055967 26974 raft_consensus.cc:1226] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 4. Current term is 5. Ops: []
I20260807 11:11:02.050606 30108 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:02.046569 30123 raft_consensus.cc:2781] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 11:11:02.117944 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:02.054978 30225 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:02.058017 26978 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 467) took 8188 ms. Trace:
I20260807 11:11:02.077358 29798 raft_consensus.cc:686] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:02.064932 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 10 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:02.097997 30216 raft_consensus.cc:686] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:02.128561 25752 leader_election.cc:302] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:11:02.101768 30214 log.cc:924] Time spent T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.242s user 0.002s sys 0.000s
I20260807 11:11:02.058889 29765 raft_consensus.cc:686] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:02.068231 30173 leader_election.cc:288] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:01.986407 29917 consensus_queue.cc:237] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:02.135010 25752 leader_election.cc:302] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:02.009315 29847 consensus_queue.cc:1035] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.095s
W20260807 11:11:02.127166 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 79144.4us
I20260807 11:11:02.137784 25750 leader_election.cc:302] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:11:02.126610 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:02.105192 30218 raft_consensus.cc:686] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:02.136193 25797 raft_consensus.cc:2454] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:11:02.133257 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:02.131801 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1785) took 2236 ms. Trace:
I20260807 11:11:02.104116 30171 leader_election.cc:288] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:02.140854 25869 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:02.143496 25869 leader_election.cc:302] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:02.145275 25796 raft_consensus.cc:2454] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:11:02.148615 25798 raft_consensus.cc:2454] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 9.
I20260807 11:11:02.150013 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:02.154110 29939 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:02.158345 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1872) took 1197 ms. Trace:
I20260807 11:11:02.131871 29972 raft_consensus.cc:2781] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:11:02.158854 26038 rpcz_store.cc:276] 0807 11:11:00.960471 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:01.942804 (+982333us) service_pool.cc:227] Handling call
0807 11:11:01.972879 (+ 30075us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:01.973223 (+ 344us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:01.973250 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:01.973680 (+ 430us) log.cc:842] Serialized 26 byte log entry
0807 11:11:02.109980 (+136300us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:11:02.110054 (+ 74us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:02.110152 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:02.158117 (+ 47965us) raft_consensus.cc:1671] finished
0807 11:11:02.158137 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:02.158274 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3773,"raft.queue_time_us":2264,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":4807,"thread_start_us":122402,"threads_started":1,"wal-append.queue_time_us":179910}
I20260807 11:11:02.136683 30123 raft_consensus.cc:686] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 7 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:02.177747 30123 consensus_queue.cc:237] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:02.139513 30178 raft_consensus.cc:2726] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 11:11:02.139789 30154 raft_consensus.cc:2781] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:11:02.184918 29875 consensus_queue.cc:1046] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260807 11:11:02.185964 30176 log.cc:924] Time spent T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260807 11:11:02.125978 30129 consensus_queue.cc:237] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:02.187036 30227 raft_consensus.cc:3037] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:02.190852 29765 consensus_queue.cc:237] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:02.191438 30182 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:02.160318 29883 raft_consensus.cc:2726] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 11:11:02.193186 30173 raft_consensus.cc:482] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:02.193672 30173 raft_consensus.cc:504] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:02.127477 30225 raft_consensus.cc:504] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:02.195209 29798 consensus_queue.cc:237] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:02.196686 30225 leader_election.cc:288] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:11:02.200237 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 83846.6us
I20260807 11:11:02.180286 29893 raft_consensus.cc:504] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:02.201020 30094 consensus_queue.cc:237] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:02.202016 25981 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:02.202813 30112 raft_consensus.cc:2726] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:02.166925 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:02.204007 25806 raft_consensus.cc:3037] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:11:02.134181 29857 raft_consensus.cc:2781] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260807 11:11:02.210027 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 57928.8us
I20260807 11:11:02.171469 30218 consensus_queue.cc:237] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:02.173427 25786 raft_consensus.cc:3037] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:11:02.175114 26974 raft_consensus.cc:1261] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260807 11:11:02.175868 30216 consensus_queue.cc:237] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:11:02.207088 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 67038.1us
I20260807 11:11:02.169142 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:02.308194 25798 raft_consensus.cc:3037] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:02.140419 29966 raft_consensus.cc:2781] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:02.124133 30169 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:02.312101 25752 leader_election.cc:302] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:11:02.289333 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 139439us
I20260807 11:11:02.305192 25869 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:11:02.223852 26035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1870) took 1276 ms. Trace:
I20260807 11:11:02.158296 30153 raft_consensus.cc:2781] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:11:02.286041 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:02.212379 29857 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:02.314967 26035 rpcz_store.cc:276] 0807 11:11:00.947127 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:01.891610 (+944483us) service_pool.cc:227] Handling call
0807 11:11:01.892092 (+ 482us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:01.908110 (+ 16018us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:01.908154 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:01.908716 (+ 562us) log.cc:842] Serialized 26 byte log entry
0807 11:11:02.000138 (+ 91422us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:11:02.000174 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:02.000240 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:02.223619 (+223379us) raft_consensus.cc:1671] finished
0807 11:11:02.223641 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:02.223784 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":178,"raft.queue_time_us":6309,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":4032,"thread_start_us":63674,"threads_started":1,"wal-append.queue_time_us":207274}
W20260807 11:11:02.412711 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 107639us
I20260807 11:11:02.224817 29972 raft_consensus.cc:686] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:02.411545 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:02.420442 25869 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:02.224341 30154 raft_consensus.cc:686] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:02.491786 26938 leader_election.cc:302] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:02.496485 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:11:02.496897 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 65312.4us
I20260807 11:11:02.318007 29966 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:02.495421 25798 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:11:02.509124 25981 leader_election.cc:302] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:02.422454 29939 raft_consensus.cc:2781] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:11:02.520975 30171 raft_consensus.cc:2726] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 11:11:02.521581 25918 raft_consensus.cc:1261] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260807 11:11:02.521960 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:02.520905 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:02.523202 25925 raft_consensus.cc:2396] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 11:11:02.514770 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:02.524194 26038 raft_consensus.cc:2396] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 7 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 11:11:02.524628 25931 raft_consensus.cc:1226] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [term 5 LEADER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: []
I20260807 11:11:02.525061 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:02.525897 25920 raft_consensus.cc:2379] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 6: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:11:02.518054 26938 leader_election.cc:302] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:02.422087 29857 raft_consensus.cc:3037] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:02.526763 29972 consensus_queue.cc:237] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:02.529217 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:02.529503 29692 raft_consensus.cc:3032] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:11:02.529937 29692 raft_consensus.cc:729] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:02.529543 25931 raft_consensus.cc:1261] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 11:11:02.531308 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:02.531548 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:02.529891 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:02.530346 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:11:02.539355 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 122046us
I20260807 11:11:02.540313 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:02.544116 25920 raft_consensus.cc:2379] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 6: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:11:02.531006 29692 consensus_queue.cc:260] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:02.485536 30112 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
W20260807 11:11:02.545120 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 124745us
I20260807 11:11:02.545197 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:02.516108 29966 raft_consensus.cc:3037] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:02.520948 30131 raft_consensus.cc:2781] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:11:02.512171 30129 raft_consensus.cc:2726] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 11:11:02.539048 29950 raft_consensus.cc:2741] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20260807 11:11:02.628226 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:02.704619 25918 raft_consensus.cc:3037] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:11:02.635860 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:02.536196 29939 raft_consensus.cc:686] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
W20260807 11:11:02.641568 30162 consensus_meta.cc:287] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.951s user 0.013s sys 0.000s
W20260807 11:11:02.682298 30243 log.cc:924] Time spent T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.366s user 0.000s sys 0.001s
W20260807 11:11:02.684013 30237 log.cc:924] Time spent T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.370s user 0.001s sys 0.000s
I20260807 11:11:02.705777 25916 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:11:02.510938 29970 consensus_queue.cc:1046] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.118s
I20260807 11:11:02.123898 26978 rpcz_store.cc:276] 0807 11:10:53.869268 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.109432 (+240164us) service_pool.cc:227] Handling call
0807 11:10:54.524537 (+415105us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:10:54.670740 (+146203us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:10:54.674633 (+ 3893us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:10:54.758306 (+ 83673us) log.cc:842] Serialized 26 byte log entry
0807 11:10:55.319912 (+561606us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:10:55.329340 (+ 9428us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:10:55.337431 (+ 8091us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:10:59.973476 (+4636045us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:11:00.353986 (+380510us) raft_consensus.cc:1671] finished
0807 11:11:00.354011 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:02.057704 (+1703693us) spinlock_profiling.cc:243] Waited 391 ms on lock 0x7b54006edf50. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:02.057853 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":114742,"raft.queue_time_us":173159,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":10495,"spinlock_wait_cycles":923329280,"thread_start_us":453199,"threads_started":1,"wal-append.queue_time_us":765592,"wal-append.run_cpu_time_us":73085,"wal-append.run_wall_time_us":5805018}
W20260807 11:11:02.687391 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 263728us
I20260807 11:11:02.694974 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:02.700517 25932 raft_consensus.cc:1226] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 5. Current term is 6. Ops: [4.6-5.7]
I20260807 11:11:02.432197 29851 consensus_queue.cc:1035] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.108s
I20260807 11:11:02.540417 30231 raft_consensus.cc:2726] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 11:11:02.544858 30182 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:02.624384 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:11:02.711086 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 80433us
I20260807 11:11:02.719306 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 478) took 2925 ms. Trace:
W20260807 11:11:02.720520 25786 consensus_meta.cc:287] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.506s user 0.010s sys 0.000s
I20260807 11:11:02.635878 30090 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:02.633003 30225 consensus_queue.cc:1035] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.110s
I20260807 11:11:02.676666 30112 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:03.120558 26980 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50942 (request call id 448) took 4927 ms. Trace:
I20260807 11:11:03.122836 26980 rpcz_store.cc:276] 0807 11:10:58.196614 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:59.286737 (+1090123us) service_pool.cc:227] Handling call
0807 11:10:59.304997 (+ 18260us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:00.536422 (+1231425us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:03.120215 (+2583793us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54005dded4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:03.120461 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2814253824}
I20260807 11:11:02.731796 25931 raft_consensus.cc:1226] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 7 LEADER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260807 11:11:01.468914 26042 rpcz_store.cc:276] 0807 11:10:59.358455 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:00.940967 (+1582512us) service_pool.cc:227] Handling call
0807 11:11:00.965898 (+ 24931us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:01.013019 (+ 47121us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 11:11:01.341062 (+328043us) log.cc:842] Serialized 14 byte log entry
0807 11:11:01.465435 (+124373us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:01.465510 (+ 75us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:01.465579 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:01.465668 (+ 89us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:01.465827 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16284,"prepare.queue_time_us":271106,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":6119,"raft.queue_time_us":18723,"spinlock_wait_cycles":825728,"thread_start_us":346802,"threads_started":2}
I20260807 11:11:03.031137 25796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1813) took 1071 ms. Trace:
I20260807 11:11:03.126552 25796 rpcz_store.cc:276] 0807 11:11:01.960149 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:01.990677 (+ 30528us) service_pool.cc:227] Handling call
0807 11:11:03.030895 (+1040218us) spinlock_profiling.cc:243] Waited 67.5 ms on lock 0x7b540014e1d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:03.031056 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":155322112}
W20260807 11:11:03.107311 25806 consensus_meta.cc:287] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.891s user 0.008s sys 0.004s
I20260807 11:11:03.140623 25806 raft_consensus.cc:2454] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:11:03.141543 25806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59118 (request call id 391) took 1069 ms. Trace:
I20260807 11:11:03.142050 25806 rpcz_store.cc:276] 0807 11:11:02.072095 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:02.138955 (+ 66860us) service_pool.cc:227] Handling call
0807 11:11:03.141474 (+1002519us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:03.121946 25799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1802) took 2520 ms. Trace:
I20260807 11:11:03.114480 25801 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
W20260807 11:11:03.144245 25801 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59118 (request call id 381) took 2312 ms (client timeout 2084 ms). Trace:
W20260807 11:11:03.144639 25801 rpcz_store.cc:269] 0807 11:11:00.832260 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:00.885185 (+ 52925us) service_pool.cc:227] Handling call
0807 11:11:03.114270 (+2229085us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b54002667a8. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc563d434 00007f3fc4606437 00007f3fc4605bc8 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:03.144176 (+ 29906us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24227584}
I20260807 11:11:02.908345 25918 raft_consensus.cc:2454] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:11:03.143651 25799 rpcz_store.cc:276] 0807 11:11:00.601771 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:00.613803 (+ 12032us) service_pool.cc:227] Handling call
0807 11:11:00.644619 (+ 30816us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:00.774193 (+129574us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 11:11:01.021062 (+246869us) log.cc:842] Serialized 14 byte log entry
0807 11:11:02.121565 (+1100503us) log.cc:842] Serialized 14 byte log entry
0807 11:11:02.156461 (+ 34896us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:02.156517 (+ 56us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:02.156557 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:03.121734 (+965177us) spinlock_profiling.cc:243] Waited 98.5 ms on lock 0x7b540014f0d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:03.121782 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:03.121894 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":588145,"prepare.queue_time_us":310302,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":79,"raft.queue_time_us":600750,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":82,"spinlock_wait_cycles":243931776,"thread_start_us":726845,"threads_started":3,"wal-append.queue_time_us":380889}
W20260807 11:11:03.182715 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 60028us
I20260807 11:11:03.183426 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:03.022452 30169 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:03.188174 25917 raft_consensus.cc:1261] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260807 11:11:02.914067 30237 log.cc:924] Time spent T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.134s user 0.001s sys 0.000s
I20260807 11:11:03.142082 26041 rpcz_store.cc:276] 0807 11:10:59.793868 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:01.860866 (+2066998us) service_pool.cc:227] Handling call
0807 11:11:01.872106 (+ 11240us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:01.880533 (+ 8427us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:02.718583 (+838050us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b54000bb150. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:02.719158 (+ 575us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25061632}
W20260807 11:11:03.115727 30257 log.cc:924] Time spent T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.054s user 0.000s sys 0.001s
I20260807 11:11:03.197135 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:02.729439 29950 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:11:02.972117 30249 log.cc:924] Time spent T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.229s user 0.000s sys 0.001s
I20260807 11:11:03.201115 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:11:02.958422 30113 consensus_meta.cc:287] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.845s user 0.008s sys 0.002s
W20260807 11:11:03.074517 30227 consensus_meta.cc:287] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.887s user 0.011s sys 0.000s
W20260807 11:11:03.065493 30188 log.cc:924] Time spent T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 1.409s user 0.004s sys 0.000s
I20260807 11:11:02.714884 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:11:02.969555 30105 consensus_meta.cc:287] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 1.290s user 0.013s sys 0.000s
I20260807 11:11:03.199373 29692 raft_consensus.cc:3037] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
W20260807 11:11:03.062474 30250 log.cc:924] Time spent T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.134s user 0.000s sys 0.001s
W20260807 11:11:03.065541 29857 consensus_meta.cc:287] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.539s user 0.008s sys 0.001s
I20260807 11:11:03.205590 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:02.794853 30173 leader_election.cc:288] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:02.779246 29898 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:03.213080 26042 raft_consensus.cc:2379] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 6: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:11:03.214206 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:02.920511 30153 raft_consensus.cc:686] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:02.729509 30162 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:03.252467 25869 leader_election.cc:302] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:03.216730 25981 leader_election.cc:302] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:02.847609 29893 leader_election.cc:288] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:03.219007 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:03.218989 30129 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:02.939759 26976 raft_consensus.cc:1226] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 LEADER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 8. Current term is 9. Ops: []
W20260807 11:11:03.220018 30123 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.016s user 0.016s sys 0.069s
I20260807 11:11:03.205345 29875 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:03.243275 25931 raft_consensus.cc:2379] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 9: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:11:03.249776 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:11:03.271788 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 364474us
I20260807 11:11:03.247995 29976 raft_consensus.cc:482] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:03.258544 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:03.222221 30227 raft_consensus.cc:2726] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:03.208711 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:03.235477 30218 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:03.267103 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:03.238502 30030 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
W20260807 11:11:03.222190 30255 log.cc:924] Time spent T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.177s user 0.001s sys 0.000s
I20260807 11:11:03.184132 30090 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:03.247660 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:03.021992 29939 consensus_queue.cc:237] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:03.277937 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:03.216248 29950 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:03.273049 29893 raft_consensus.cc:2726] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:03.297421 30169 raft_consensus.cc:2726] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:03.298280 29950 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:03.283872 29976 raft_consensus.cc:504] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:03.242514 30108 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.007s sys 0.010s
W20260807 11:11:03.299173 30108 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.592s user 0.007s sys 0.010s
I20260807 11:11:03.300104 29976 leader_election.cc:288] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:03.286227 29851 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:03.301192 29851 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:03.303283 29851 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:03.237738 30105 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:03.284714 25750 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.000s sys 0.024s
W20260807 11:11:03.310869 25750 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.597s user 0.000s sys 0.024s
I20260807 11:11:03.335852 25803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1815) took 1375 ms. Trace:
W20260807 11:11:03.270794 30123 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.067s user 0.016s sys 0.070s
I20260807 11:11:02.847381 30154 consensus_queue.cc:237] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:03.248205 29970 consensus_queue.cc:1046] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20260807 11:11:03.222759 30113 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:03.242365 29857 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:03.294286 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:03.279305 26041 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
W20260807 11:11:03.280328 29765 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.533s user 0.001s sys 0.020s
I20260807 11:11:03.317166 26936 leader_election.cc:302] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters: b22d69db45194908ad3d345eba9badd9
W20260807 11:11:03.311198 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 597769us
I20260807 11:11:03.337231 25803 rpcz_store.cc:276] 0807 11:11:01.960727 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:02.208986 (+248259us) service_pool.cc:227] Handling call
0807 11:11:02.209407 (+ 421us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:02.289270 (+ 79863us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:02.289303 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:02.314868 (+ 25565us) log.cc:842] Serialized 26 byte log entry
0807 11:11:02.708588 (+393720us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:11:02.708654 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:02.708780 (+ 126us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:03.335581 (+626801us) raft_consensus.cc:1671] finished
0807 11:11:03.335609 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:03.335776 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14979,"raft.queue_time_us":15336,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":18893,"thread_start_us":200813,"threads_started":1,"wal-append.queue_time_us":541674}
I20260807 11:11:03.177847 30131 raft_consensus.cc:686] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:03.256409 29898 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:03.278251 29875 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:03.274564 30129 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:04.062439 26975 raft_consensus.cc:3037] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Advancing to term 7
W20260807 11:11:04.063347 29966 consensus_meta.cc:287] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 1.385s user 0.012s sys 0.000s
I20260807 11:11:04.063863 29966 raft_consensus.cc:504] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:03.646642 25800 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.007s sys 0.017s
W20260807 11:11:04.065229 25800 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.921s user 0.015s sys 0.029s
I20260807 11:11:03.721903 25786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1742) took 8974 ms. Trace:
I20260807 11:11:03.559446 25799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1819) took 1596 ms. Trace:
W20260807 11:11:03.914479 25752 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.704s user 0.014s sys 0.023s
I20260807 11:11:04.067301 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:04.071472 25918 raft_consensus.cc:2379] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 7: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:11:03.392792 25920 raft_consensus.cc:1226] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 4. Current term is 5. Ops: []
I20260807 11:11:03.378270 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:03.393682 25916 raft_consensus.cc:1226] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:04.066888 25799 rpcz_store.cc:276] 0807 11:11:01.962721 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:03.183266 (+1220545us) service_pool.cc:227] Handling call
0807 11:11:03.201210 (+ 17944us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:03.201418 (+ 208us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 11:11:03.221438 (+ 20020us) log.cc:842] Serialized 14 byte log entry
0807 11:11:03.558838 (+337400us) log.cc:842] Serialized 14 byte log entry
0807 11:11:03.559061 (+ 223us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:03.559127 (+ 66us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:03.559176 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:03.559255 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:03.559384 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":57091,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":10318,"raft.queue_time_us":53885,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":87778,"spinlock_wait_cycles":298348800,"thread_start_us":299578,"threads_started":1}
I20260807 11:11:03.340636 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:04.081791 26042 raft_consensus.cc:2396] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260807 11:11:04.088464 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 796822us
I20260807 11:11:03.457557 25932 raft_consensus.cc:1261] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260807 11:11:04.067963 25752 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.858s user 0.014s sys 0.030s
W20260807 11:11:04.068164 26938 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.613s user 0.005s sys 0.010s
I20260807 11:11:03.764178 25925 raft_consensus.cc:2454] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 7.
W20260807 11:11:04.096994 26938 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.642s user 0.006s sys 0.010s
I20260807 11:11:04.101632 26980 raft_consensus.cc:2379] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 6: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:11:04.066463 25786 rpcz_store.cc:276] 0807 11:10:54.727616 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:54.727757 (+ 141us) service_pool.cc:227] Handling call
0807 11:10:54.750286 (+ 22529us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:10:58.149334 (+3399048us) spinlock_profiling.cc:243] Waited 3.05 s on lock 0x7b5800725fb0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf63ff63 00007f3fbf6a3aba 00007f3fbf6bb3af 00007f3fbf6bcb4e 00007f3fbf6c6c89 00007f3fbf6c7556 00007f3fbf6c3edf 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684
0807 11:11:02.133932 (+3984598us) spinlock_profiling.cc:243] Waited 2.54 s on lock 0x7b54001e9720. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf65240d 00007f3fbf640050 00007f3fbf6a3aba 00007f3fbf6bb3af 00007f3fbf6bcb4e 00007f3fbf6c6c89 00007f3fbf6c7556 00007f3fbf6c3edf 00007f3fbf6c399e 00007f3fc46052e9
0807 11:11:03.648605 (+1514673us) raft_consensus.cc:1520] Early marking committed up to index 1
0807 11:11:03.648650 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:03.702006 (+ 53356us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:11:03.702102 (+ 96us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:03.702214 (+ 112us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:03.702413 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14325,"raft.queue_time_us":107654,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":79,"spinlock_wait_cycles":12846505984,"thread_start_us":349133,"threads_started":1}
W20260807 11:11:03.517971 25988 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.647s user 0.011s sys 0.008s
W20260807 11:11:04.130882 25988 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.510s user 0.014s sys 0.018s
W20260807 11:11:04.130725 25750 connection.cc:232] RPC call timeout handler was delayed by -1.6477487680967897s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1533s for a response
W20260807 11:11:04.131910 25750 connection.cc:232] RPC call timeout handler was delayed by -1.4398487680591643s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1764s for a response
W20260807 11:11:04.132421 25750 connection.cc:232] RPC call timeout handler was delayed by -1.2652487680315971s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1958s for a response
W20260807 11:11:04.132663 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 932335us
I20260807 11:11:04.133682 26980 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 485) took 1340 ms. Trace:
I20260807 11:11:04.134315 26980 rpcz_store.cc:276] 0807 11:11:02.793627 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:03.307094 (+513467us) service_pool.cc:227] Handling call
0807 11:11:04.133600 (+826506us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:04.135227 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:04.149376 26041 raft_consensus.cc:1226] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: []
I20260807 11:11:04.150302 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1887) took 2817 ms. Trace:
I20260807 11:11:04.150717 26041 rpcz_store.cc:276] 0807 11:11:01.333103 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.130539 (+2797436us) service_pool.cc:227] Handling call
0807 11:11:04.149122 (+ 18583us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.149984 (+ 862us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.150235 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:11:04.155390 26039 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43246 (request call id 480) took 4261 ms (client timeout 1513 ms). Trace:
W20260807 11:11:03.680823 26974 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.016s sys 0.003s
I20260807 11:11:04.156852 25797 raft_consensus.cc:2379] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 8: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
W20260807 11:11:03.479167 30265 log.cc:924] Time spent T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.144s user 0.000s sys 0.000s
I20260807 11:11:02.426024 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:11:04.170161 25919 raft_consensus.cc:1226] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: []
W20260807 11:11:04.155823 26039 rpcz_store.cc:269] 0807 11:10:59.870909 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:59.915383 (+ 44474us) service_pool.cc:227] Handling call
0807 11:11:04.075060 (+4159677us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b54003732d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:04.132602 (+ 57542us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3814370048}
I20260807 11:11:04.159016 26041 raft_consensus.cc:1261] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 11:11:04.172401 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1890) took 2818 ms. Trace:
W20260807 11:11:03.521113 30272 log.cc:924] Time spent T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260807 11:11:03.849682 30225 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:04.228044 26041 rpcz_store.cc:276] 0807 11:11:01.353756 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.151767 (+2798011us) service_pool.cc:227] Handling call
0807 11:11:04.158533 (+ 6766us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.172016 (+ 13483us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.172318 (+ 302us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:03.489053 30288 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:04.161721 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
W20260807 11:11:04.166003 26035 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.207s user 0.008s sys 0.012s
W20260807 11:11:04.233132 26035 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.274s user 0.008s sys 0.012s
I20260807 11:11:04.167627 25786 pending_rounds.cc:76] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815: Aborting all ops after (but not including) 6
I20260807 11:11:04.234510 25786 pending_rounds.cc:97] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20260807 11:11:04.234862 25786 raft_consensus.cc:2866] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:11:03.357373 30227 raft_consensus.cc:482] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:03.481105 29977 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:03.669401 29950 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:03.959461 30266 raft_consensus.cc:482] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:04.245355 30266 raft_consensus.cc:504] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:04.246665 29875 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195)
I20260807 11:11:04.247802 30266 leader_election.cc:288] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:04.060948 29995 raft_consensus.cc:482] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:04.248521 29995 raft_consensus.cc:504] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:03.849892 29851 raft_consensus.cc:482] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:04.249820 29851 raft_consensus.cc:504] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:04.251834 29851 leader_election.cc:288] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:04.172631 26042 raft_consensus.cc:1226] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 9 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 8. Current term is 9. Ops: []
I20260807 11:11:04.252578 29898 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:04.252813 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1888) took 2908 ms. Trace:
I20260807 11:11:04.253250 26042 rpcz_store.cc:276] 0807 11:11:01.344532 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.135411 (+2790879us) service_pool.cc:227] Handling call
0807 11:11:04.172418 (+ 37007us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.252550 (+ 80132us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.252740 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:04.254495 26042 raft_consensus.cc:1226] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 5. Current term is 6. Ops: []
I20260807 11:11:04.255043 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1790) took 2801 ms. Trace:
I20260807 11:11:04.255359 26042 rpcz_store.cc:276] 0807 11:11:01.453931 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.254008 (+2800077us) service_pool.cc:227] Handling call
0807 11:11:04.254321 (+ 313us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.254840 (+ 519us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.254994 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:04.217059 25981 leader_election.cc:302] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:04.267000 25750 leader_election.cc:302] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:04.128875 30216 raft_consensus.cc:2781] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:11:03.336217 26978 raft_consensus.cc:2454] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 7.
I20260807 11:11:03.741540 30169 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:03.286801 30218 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:04.235633 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1788) took 2818 ms. Trace:
I20260807 11:11:04.231845 26039 raft_consensus.cc:1226] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7. Ops: [5.4-6.5]
I20260807 11:11:03.559350 29972 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
W20260807 11:11:04.162696 26974 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.041s user 0.021s sys 0.007s
I20260807 11:11:03.309424 30030 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:03.517603 30256 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:04.256315 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:04.257748 29977 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:04.226101 30090 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:04.258467 30094 raft_consensus.cc:3032] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:11:04.259092 29950 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:02.140036 26034 rpcz_store.cc:276] 0807 11:10:59.894964 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:01.861281 (+1966317us) service_pool.cc:227] Handling call
0807 11:11:01.907002 (+ 45721us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:01.907258 (+ 256us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 11:11:01.919044 (+ 11786us) log.cc:842] Serialized 14 byte log entry
0807 11:11:02.129689 (+210645us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:02.131517 (+ 1828us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:02.131567 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:02.131628 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:02.131735 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":11624,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":27,"raft.queue_time_us":5312,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":5868,"thread_start_us":204994,"threads_started":1}
I20260807 11:11:04.239589 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:11:04.259949 30287 log.cc:924] Time spent T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.467s user 0.000s sys 0.001s
W20260807 11:11:04.260712 30290 log.cc:924] Time spent T cccdf448d32448deae255de6bf7b4125 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.194s user 0.000s sys 0.001s
I20260807 11:11:04.260936 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
W20260807 11:11:04.262413 26936 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.120.129:40195 timed out after 1.682s (SENT) [suppressed 10 similar messages]
I20260807 11:11:04.262780 25796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1820) took 2299 ms. Trace:
I20260807 11:11:04.263170 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 484) took 3110 ms. Trace:
I20260807 11:11:04.263269 30102 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:04.167949 30105 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:04.358173 26034 raft_consensus.cc:1226] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 5. Current term is 6. Ops: []
I20260807 11:11:04.269346 26042 raft_consensus.cc:1261] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260807 11:11:04.155982 29765 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:04.279229 26041 rpcz_store.cc:276] 0807 11:11:01.417539 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.230129 (+2812590us) service_pool.cc:227] Handling call
0807 11:11:04.234630 (+ 4501us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.234982 (+ 352us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:04.235020 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:04.235072 (+ 52us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:04.235117 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.235309 (+ 192us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:04.235553 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:04.279670 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1891) took 2910 ms. Trace:
I20260807 11:11:04.289111 30216 raft_consensus.cc:686] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:04.361765 26039 rpcz_store.cc:276] 0807 11:11:01.369567 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.171310 (+2801743us) service_pool.cc:227] Handling call
0807 11:11:04.231517 (+ 60207us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:11:04.279368 (+ 47851us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.279610 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:04.168386 30112 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:04.225481 30162 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 8 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:04.294721 25750 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:04.299383 30094 raft_consensus.cc:729] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:04.304694 25796 rpcz_store.cc:276] 0807 11:11:01.963001 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:03.207298 (+1244297us) service_pool.cc:227] Handling call
0807 11:11:03.207735 (+ 437us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:03.207957 (+ 222us) raft_consensus.cc:1520] Early marking committed up to index 1
0807 11:11:03.207980 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:03.208449 (+ 469us) log.cc:842] Serialized 26 byte log entry
0807 11:11:03.707572 (+499123us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:11:03.707622 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:03.707734 (+ 112us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:04.262559 (+554825us) raft_consensus.cc:1671] finished
0807 11:11:04.262586 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:04.262727 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":451551,"threads_started":1,"wal-append.queue_time_us":936343}
I20260807 11:11:04.370240 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:04.311403 25931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1795) took 1032 ms. Trace:
I20260807 11:11:04.338955 26979 raft_consensus.cc:2379] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 6: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:11:04.304800 26038 rpcz_store.cc:276] 0807 11:11:01.153112 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:02.524725 (+1371613us) service_pool.cc:227] Handling call
0807 11:11:02.525063 (+ 338us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:02.539509 (+ 14446us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:02.539551 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:02.637246 (+ 97695us) log.cc:842] Serialized 26 byte log entry
0807 11:11:03.334813 (+697567us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:03.334870 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:03.334963 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:04.262975 (+928012us) raft_consensus.cc:1671] finished
0807 11:11:04.262994 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:04.263106 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14053,"raft.queue_time_us":14661,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":5251,"thread_start_us":485562,"threads_started":1,"wal-append.queue_time_us":1481866}
I20260807 11:11:04.371675 26979 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 483) took 1661 ms. Trace:
I20260807 11:11:04.372188 26979 rpcz_store.cc:276] 0807 11:11:02.710345 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:02.710510 (+ 165us) service_pool.cc:227] Handling call
0807 11:11:04.260758 (+1550248us) spinlock_profiling.cc:243] Waited 49.8 ms on lock 0x7b5800000f98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc46061b0 00007f3fc4605a94 00007f3fbdcedc04 00007f3fbdcedb92
0807 11:11:04.371599 (+110841us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":114565376}
I20260807 11:11:04.371315 25931 rpcz_store.cc:276] 0807 11:11:03.279421 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:03.283495 (+ 4074us) service_pool.cc:227] Handling call
0807 11:11:03.284913 (+ 1418us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:03.285114 (+ 201us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 11:11:03.758179 (+473065us) log.cc:842] Serialized 14 byte log entry
0807 11:11:04.304043 (+545864us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:04.311076 (+ 7033us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:04.311147 (+ 71us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.311223 (+ 76us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:04.311351 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":971991,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"thread_start_us":943546,"threads_started":3,"wal-append.queue_time_us":519391}
I20260807 11:11:03.490474 30153 consensus_queue.cc:237] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:04.315763 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 142465us
I20260807 11:11:04.283116 29966 leader_election.cc:288] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:04.359377 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1791) took 2885 ms. Trace:
I20260807 11:11:04.363059 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1792) took 2817 ms. Trace:
I20260807 11:11:04.387169 26034 rpcz_store.cc:276] 0807 11:11:01.545705 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.301022 (+2755317us) service_pool.cc:227] Handling call
0807 11:11:04.352320 (+ 51298us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.362735 (+ 10415us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.362987 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:04.297689 29977 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:04.296618 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:04.238775 30225 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:04.365621 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:11:04.393010 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 98389.2us
I20260807 11:11:04.393481 26041 raft_consensus.cc:2454] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:11:04.393980 25750 leader_election.cc:302] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:11:04.298048 26978 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 488) took 1382 ms (client timeout 1544 ms). Trace:
W20260807 11:11:04.395037 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 325662us
I20260807 11:11:04.381199 26979 raft_consensus.cc:1261] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260807 11:11:04.381783 30216 consensus_queue.cc:237] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:04.382916 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:04.401283 26039 raft_consensus.cc:2379] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 4: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
W20260807 11:11:04.383451 26939 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.618s user 0.016s sys 0.021s
W20260807 11:11:04.401711 26939 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.709s user 0.017s sys 0.021s
I20260807 11:11:04.401938 26979 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50942 (request call id 476) took 1216 ms. Trace:
I20260807 11:11:04.402377 26979 rpcz_store.cc:276] 0807 11:11:03.181505 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.373543 (+1192038us) service_pool.cc:227] Handling call
0807 11:11:04.377589 (+ 4046us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.397317 (+ 19728us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.397562 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:04.243939 30288 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:04.389999 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:04.395459 25752 leader_election.cc:302] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:04.408241 26038 raft_consensus.cc:3037] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:04.299409 26980 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 489) took 1385 ms. Trace:
I20260807 11:11:04.396293 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:04.266337 30227 raft_consensus.cc:504] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:04.298058 29898 consensus_queue.cc:1046] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.069s
I20260807 11:11:04.289875 30169 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:04.400321 26041 raft_consensus.cc:1261] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260807 11:11:04.290426 29972 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:04.388744 26042 rpcz_store.cc:276] 0807 11:11:01.474479 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.267656 (+2793177us) service_pool.cc:227] Handling call
0807 11:11:04.268973 (+ 1317us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.359019 (+ 90046us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.359304 (+ 285us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:04.408905 26980 rpcz_store.cc:276] 0807 11:11:02.913584 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.166961 (+1253377us) service_pool.cc:227] Handling call
0807 11:11:04.299305 (+132344us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:04.409425 25796 raft_consensus.cc:2454] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 8.
I20260807 11:11:04.412590 25981 leader_election.cc:302] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:04.399137 28964 raft_consensus.cc:2726] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:11:04.414572 29802 raft_consensus.cc:2781] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:04.402221 29966 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:04.416172 26039 raft_consensus.cc:1226] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4. Ops: []
I20260807 11:11:04.418674 26041 pending_rounds.cc:76] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9: Aborting all ops after (but not including) 2
I20260807 11:11:04.542703 26041 pending_rounds.cc:97] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260807 11:11:04.543016 26041 raft_consensus.cc:2866] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:11:04.361726 30297 consensus_queue.cc:1035] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.175s
I20260807 11:11:04.417671 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1793) took 1181 ms. Trace:
I20260807 11:11:04.298292 30256 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:04.365473 30102 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:04.361340 29851 consensus_queue.cc:1046] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.045s
I20260807 11:11:04.444854 29799 raft_consensus.cc:2726] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:04.531343 26980 raft_consensus.cc:1226] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 4. Current term is 5. Ops: []
I20260807 11:11:04.410995 30296 raft_consensus.cc:2726] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260807 11:11:04.399515 26978 rpcz_store.cc:269] 0807 11:11:02.901096 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:03.188469 (+287373us) service_pool.cc:227] Handling call
0807 11:11:04.283084 (+1094615us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:04.536103 26042 raft_consensus.cc:1226] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 5. Current term is 6. Ops: []
I20260807 11:11:04.418119 26034 raft_consensus.cc:1261] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 11:11:04.546373 25796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1824) took 2554 ms. Trace:
I20260807 11:11:04.304901 30131 consensus_queue.cc:237] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:04.386749 29765 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:04.564824 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1802) took 2906 ms. Trace:
I20260807 11:11:04.644021 26039 rpcz_store.cc:276] 0807 11:11:01.658526 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.403831 (+2745305us) service_pool.cc:227] Handling call
0807 11:11:04.415968 (+ 12137us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.564470 (+148502us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.564735 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:04.544260 25799 raft_consensus.cc:2454] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
W20260807 11:11:04.644718 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 382307us
W20260807 11:11:04.642822 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 244871us
I20260807 11:11:04.601011 26980 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50942 (request call id 478) took 1123 ms. Trace:
I20260807 11:11:04.645924 26980 rpcz_store.cc:276] 0807 11:11:03.477972 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.501355 (+1023383us) service_pool.cc:227] Handling call
0807 11:11:04.531124 (+ 29769us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.600687 (+ 69563us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.600947 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:11:04.646471 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 252562us
W20260807 11:11:04.613790 30295 log.cc:924] Time spent T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
W20260807 11:11:04.614763 30318 log.cc:924] Time spent T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20260807 11:11:04.621208 29851 raft_consensus.cc:3032] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 8 LEADER]: Stepping down as leader of term 8
I20260807 11:11:04.622145 30227 leader_election.cc:288] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:04.623425 29950 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:04.624102 30030 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:04.624637 30327 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:04.626381 30102 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:04.629522 30225 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:04.638562 25786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1822) took 2666 ms. Trace:
I20260807 11:11:04.641582 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1798) took 2984 ms. Trace:
I20260807 11:11:04.678972 26034 rpcz_store.cc:276] 0807 11:11:01.657381 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.388209 (+2730828us) service_pool.cc:227] Handling call
0807 11:11:04.412581 (+ 24372us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.641218 (+228637us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.641510 (+ 292us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:04.641305 25796 rpcz_store.cc:276] 0807 11:11:01.991858 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.544567 (+2552709us) service_pool.cc:227] Handling call
0807 11:11:04.545510 (+ 943us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.545991 (+ 481us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:04.546037 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:04.546087 (+ 50us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:11:04.546125 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.546187 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:04.546311 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:04.678874 25786 rpcz_store.cc:276] 0807 11:11:01.972152 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.167034 (+2194882us) service_pool.cc:227] Handling call
0807 11:11:04.167423 (+ 389us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:04.243190 (+ 75767us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:04.243227 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:04.243648 (+ 421us) log.cc:842] Serialized 26 byte log entry
0807 11:11:04.501977 (+258329us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:04.502010 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.502733 (+ 723us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:04.638231 (+135498us) raft_consensus.cc:1671] finished
0807 11:11:04.638262 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:04.638480 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":214567,"threads_started":1,"wal-append.queue_time_us":339107}
I20260807 11:11:04.565944 26038 raft_consensus.cc:2454] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 7.
I20260807 11:11:04.542305 29802 raft_consensus.cc:482] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:04.688576 29802 raft_consensus.cc:3037] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
W20260807 11:11:04.689411 29692 consensus_meta.cc:287] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 1.451s user 0.013s sys 0.000s
I20260807 11:11:04.690018 29765 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:11:04.691321 26975 consensus_meta.cc:287] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.628s user 0.017s sys 0.009s
I20260807 11:11:04.691666 26975 raft_consensus.cc:2454] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 7.
I20260807 11:11:04.692008 26986 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:11:04.692243 26975 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 475) took 1511 ms. Trace:
I20260807 11:11:04.647042 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1801) took 2988 ms. Trace:
W20260807 11:11:04.693110 26986 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 487) took 1826 ms (client timeout 2258 ms). Trace:
I20260807 11:11:04.693593 25801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1818) took 2731 ms. Trace:
W20260807 11:11:04.693495 26986 rpcz_store.cc:269] 0807 11:11:02.866871 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:03.263038 (+396167us) service_pool.cc:227] Handling call
0807 11:11:04.692955 (+1429917us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b5400760554. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:04.693056 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3222731520}
I20260807 11:11:04.694262 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:04.694733 29977 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:04.695289 29977 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:04.695650 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:04.659776 29851 raft_consensus.cc:729] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:04.696705 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:04.696895 30102 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:04.697335 25932 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:04.697403 30102 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:04.697631 29851 consensus_queue.cc:260] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:04.698762 29851 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Advancing to term 9
I20260807 11:11:04.699352 30334 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:04.649331 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1803) took 2990 ms. Trace:
I20260807 11:11:04.700706 26042 rpcz_store.cc:276] 0807 11:11:01.658789 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.422619 (+2763830us) service_pool.cc:227] Handling call
0807 11:11:04.535936 (+113317us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.649074 (+113138us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.649270 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:04.651197 30129 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:04.694022 25801 rpcz_store.cc:276] 0807 11:11:01.962320 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:03.188685 (+1226365us) service_pool.cc:227] Handling call
0807 11:11:03.225105 (+ 36420us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:03.677737 (+452632us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:03.677780 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:03.712587 (+ 34807us) log.cc:842] Serialized 26 byte log entry
0807 11:11:04.340551 (+627964us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:04.412985 (+ 72434us) log.cc:842] Serialized 14 byte log entry
0807 11:11:04.413243 (+ 258us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.413324 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:04.693397 (+280073us) raft_consensus.cc:1671] finished
0807 11:11:04.693417 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:04.693539 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1485,"prepare.queue_time_us":305759,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":4380,"raft.queue_time_us":78649,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":28837,"thread_start_us":875793,"threads_started":3,"wal-append.queue_time_us":738784}
I20260807 11:11:04.704016 25801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 390) took 2644 ms. Trace:
I20260807 11:11:04.692930 26041 rpcz_store.cc:276] 0807 11:11:01.658216 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.394490 (+2736274us) service_pool.cc:227] Handling call
0807 11:11:04.400004 (+ 5514us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.646777 (+246773us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.646977 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:11:04.705595 30221 log.cc:924] Time spent T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 2.651s user 0.019s sys 0.012s
I20260807 11:11:04.692641 26975 rpcz_store.cc:276] 0807 11:11:03.181103 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:03.289224 (+108121us) service_pool.cc:227] Handling call
0807 11:11:04.692186 (+1402962us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:04.583841 29857 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:04.707017 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1786) took 4796 ms. Trace:
I20260807 11:11:04.680469 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:04.707369 26033 rpcz_store.cc:276] 0807 11:10:59.910821 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:01.861655 (+1950834us) service_pool.cc:227] Handling call
0807 11:11:01.861984 (+ 329us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:11:01.862343 (+ 359us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:01.862371 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 11:11:01.875615 (+ 13244us) log.cc:842] Serialized 52 byte log entry
0807 11:11:01.956940 (+ 81325us) raft_consensus.cc:1631] Marking committed up to 1
0807 11:11:01.956991 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:01.957080 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:04.704863 (+2747783us) spinlock_profiling.cc:243] Waited 24.7 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:11:04.706671 (+ 1808us) raft_consensus.cc:1671] finished
0807 11:11:04.706690 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:04.706953 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":98,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":16930,"spinlock_wait_cycles":56906112,"thread_start_us":42491,"threads_started":1,"wal-append.queue_time_us":149100}
I20260807 11:11:04.708611 30288 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:04.709875 30288 raft_consensus.cc:2741] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260807 11:11:04.668787 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1809) took 2864 ms. Trace:
I20260807 11:11:04.710626 25796 raft_consensus.cc:1261] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260807 11:11:04.711216 25796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1825) took 2615 ms. Trace:
I20260807 11:11:04.711838 25808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 372) took 3992 ms. Trace:
I20260807 11:11:04.711627 29977 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:04.710023 26033 raft_consensus.cc:1261] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 11:11:04.561321 30326 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:04.650704 30090 raft_consensus.cc:482] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:11:04.670044 26046 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43246 (request call id 483) took 3517 ms (client timeout 1528 ms). Trace:
I20260807 11:11:04.582899 29966 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:04.650800 29970 raft_consensus.cc:482] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:04.713562 30090 raft_consensus.cc:504] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:11:04.713726 26046 rpcz_store.cc:269] 0807 11:11:01.152785 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:01.180684 (+ 27899us) service_pool.cc:227] Handling call
0807 11:11:04.669854 (+3489170us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b54002df354. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:04.669968 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":316113792}
I20260807 11:11:04.714036 29970 raft_consensus.cc:504] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:04.682360 30329 consensus_queue.cc:1035] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.005s
I20260807 11:11:04.677948 30327 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:04.683449 30169 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:04.683696 30256 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:11:04.683962 30317 log.cc:924] Time spent T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.319s user 0.001s sys 0.000s
I20260807 11:11:04.684499 29950 raft_consensus.cc:2726] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:11:04.738483 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:04.678656 30030 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:04.763034 30030 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:04.763593 25920 raft_consensus.cc:2454] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:11:04.765043 30030 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:11:04.660900 30215 log.cc:924] Time spent T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.293s user 0.000s sys 0.001s
I20260807 11:11:04.766963 29966 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:04.768364 30090 leader_election.cc:288] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:04.713438 30326 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:04.770710 30326 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:04.744964 30113 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:04.699878 30334 raft_consensus.cc:504] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:04.704452 25801 rpcz_store.cc:276] 0807 11:11:02.059289 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.703244 (+2643955us) service_pool.cc:227] Handling call
0807 11:11:04.703545 (+ 301us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.703707 (+ 162us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:04.703730 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:04.703773 (+ 43us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:04.703817 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.703870 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:04.703966 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:04.529533 30094 consensus_queue.cc:260] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:04.710775 26039 rpcz_store.cc:276] 0807 11:11:01.804606 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.645743 (+2841137us) service_pool.cc:227] Handling call
0807 11:11:04.668268 (+ 22525us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.668443 (+ 175us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:04.668467 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:04.668505 (+ 38us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:11:04.668539 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.668590 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:04.668703 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:04.773507 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:04.774335 26039 raft_consensus.cc:2379] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 6: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:11:04.774487 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:04.681852 26038 raft_consensus.cc:1261] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 11:11:04.775332 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1810) took 2917 ms. Trace:
I20260807 11:11:04.775661 26038 rpcz_store.cc:276] 0807 11:11:01.858216 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.681275 (+2823059us) service_pool.cc:227] Handling call
0807 11:11:04.681542 (+ 267us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.775088 (+ 93546us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.775280 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:04.776292 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:04.713501 25931 raft_consensus.cc:1261] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260807 11:11:04.712298 25808 rpcz_store.cc:276] 0807 11:11:00.719645 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:01.069717 (+350072us) service_pool.cc:227] Handling call
0807 11:11:01.297037 (+227320us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:01.950893 (+653856us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:01.950924 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:01.950966 (+ 42us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:01.951000 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:01.951059 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:04.259167 (+2308108us) spinlock_profiling.cc:243] Waited 2.38 ms on lock 0x7b48001326d8. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc456b7f1 00007f3fc45628e7 00007f3fc456277a 00007f3fc465695f 00007f3fc4656627 00007f3fc466ab2c 00007f3fc466aad8 00007f3fc466aa58 00007f3fc466aa13 00007f3fc4669cbc 00007f3fc35ebbe6
0807 11:11:04.711647 (+452480us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b540018fe50. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:04.711778 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":671467,"raft.queue_time_us":689007,"raft.run_cpu_time_us":17869,"raft.run_wall_time_us":2271745,"spinlock_wait_cycles":478433408}
I20260807 11:11:04.718688 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 315) took 3896 ms. Trace:
I20260807 11:11:04.776753 26038 raft_consensus.cc:3032] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:11:04.778124 26038 raft_consensus.cc:729] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:11:04.777679 25917 rpcz_store.cc:276] 0807 11:11:00.822187 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:03.031704 (+2209517us) service_pool.cc:227] Handling call
0807 11:11:03.187852 (+156148us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:03.188561 (+ 709us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.718505 (+1529944us) spinlock_profiling.cc:243] Waited 59.2 ms on lock 0x7b54001a2050. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:04.718631 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":136225664}
I20260807 11:11:04.778666 26035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1876) took 3773 ms. Trace:
I20260807 11:11:04.779085 26038 consensus_queue.cc:260] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:04.777506 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:04.779024 26035 rpcz_store.cc:276] 0807 11:11:01.005638 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:02.535932 (+1530294us) service_pool.cc:227] Handling call
0807 11:11:02.623252 (+ 87320us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:02.623539 (+ 287us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 11:11:02.959039 (+335500us) log.cc:842] Serialized 14 byte log entry
0807 11:11:04.778251 (+1819212us) spinlock_profiling.cc:243] Waited 4.74 ms on lock 0x7b1800408460. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbf35382c 00007f3fbc77e9dc 00007f3fc55b5a39 00007f3fbc79b8bb 00007f3fbc799cbf 00007f3fbc799155 00007f3fbf6bc3d0 00007f3fbf6ce576 00007f3fbf6df0d7
0807 11:11:04.778365 (+ 114us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:04.778407 (+ 42us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:04.778456 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.778517 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:04.778615 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":661701,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":188476,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":78092,"spinlock_wait_cycles":23577344,"thread_start_us":1836209,"threads_started":3,"wal-append.queue_time_us":1426970}
I20260807 11:11:04.780246 26038 raft_consensus.cc:3037] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
W20260807 11:11:04.780540 25852 maintenance_manager.cc:445] Time spent finding best maintenance operation: real 10.438s user 0.033s sys 0.016s
I20260807 11:11:04.668095 26936 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:04.729899 25752 leader_election.cc:302] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:11:04.787791 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 58009.9us
W20260807 11:11:04.746989 25988 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.615s user 0.003s sys 0.008s
I20260807 11:11:04.747710 30288 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:04.750138 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
W20260807 11:11:04.790196 25988 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.658s user 0.003s sys 0.008s
I20260807 11:11:04.790541 25786 raft_consensus.cc:3037] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:11:04.696659 29323 leader_election.cc:288] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:04.776329 25750 leader_election.cc:302] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:11:04.765220 30302 log.cc:924] Time spent T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.392s user 0.002s sys 0.000s
I20260807 11:11:04.702390 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 492) took 2728 ms. Trace:
I20260807 11:11:04.772145 25926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 328) took 3063 ms. Trace:
I20260807 11:11:04.713379 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1812) took 2555 ms. Trace:
I20260807 11:11:04.781401 30102 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:04.782243 30105 raft_consensus.cc:3032] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 5 LEADER]: Stepping down as leader of term 5
I20260807 11:11:04.711565 25796 rpcz_store.cc:276] 0807 11:11:02.095369 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.710053 (+2614684us) service_pool.cc:227] Handling call
0807 11:11:04.710313 (+ 260us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.711013 (+ 700us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.711170 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:04.815423 25750 leader_election.cc:302] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:04.775350 25801 raft_consensus.cc:2396] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 10 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260807 11:11:04.764622 30327 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:04.784685 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:04.817243 26034 raft_consensus.cc:3037] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 8
I20260807 11:11:04.758606 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:04.806329 26033 rpcz_store.cc:276] 0807 11:11:02.157781 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.709445 (+2551664us) service_pool.cc:227] Handling call
0807 11:11:04.709697 (+ 252us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.713131 (+ 3434us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.713310 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:04.810487 25926 rpcz_store.cc:276] 0807 11:11:01.708900 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:01.709604 (+ 704us) service_pool.cc:227] Handling call
0807 11:11:01.735494 (+ 25890us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:11:01.757958 (+ 22464us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:01.758007 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 11:11:01.777060 (+ 19053us) log.cc:842] Serialized 52 byte log entry
0807 11:11:01.840012 (+ 62952us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:01.864805 (+ 24793us) log.cc:842] Serialized 14 byte log entry
0807 11:11:01.865056 (+ 251us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:01.865153 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:04.281731 (+2416578us) spinlock_profiling.cc:243] Waited 2.64 ms on lock 0x7b1800317b00. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbc79c638 00007f3fbc79fb11 00007f3fbc79fac9 00007f3fbc79fa59 00007f3fbc79fa21 00007f3fbc79ed1d 00007f3fc5a00204 00007f3fc5a00039 00007f3fbc77edc6
0807 11:11:04.771830 (+490099us) raft_consensus.cc:1671] finished
0807 11:11:04.771938 (+ 108us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:04.772080 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":24596,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":5437,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":26873,"spinlock_wait_cycles":27591296,"thread_start_us":43149,"threads_started":1,"wal-append.queue_time_us":76705}
I20260807 11:11:04.822062 26039 raft_consensus.cc:1261] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 11:11:04.811672 25869 leader_election.cc:302] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:04.804953 26042 rpcz_store.cc:276] 0807 11:11:01.973801 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.701508 (+2727707us) service_pool.cc:227] Handling call
0807 11:11:04.701855 (+ 347us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.702107 (+ 252us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:04.702134 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:04.702169 (+ 35us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:04.702195 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.702242 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:04.702338 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:04.825981 26035 raft_consensus.cc:1261] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
W20260807 11:11:04.822500 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 160732us
I20260807 11:11:04.827169 26936 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:04.827064 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:04.826540 26035 pending_rounds.cc:76] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9: Aborting all ops after (but not including) 2
I20260807 11:11:04.828792 26035 pending_rounds.cc:97] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260807 11:11:04.829371 26035 raft_consensus.cc:2866] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:11:04.832540 26035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1895) took 1645 ms. Trace:
I20260807 11:11:04.834134 26035 rpcz_store.cc:276] 0807 11:11:03.187060 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.780702 (+1593642us) service_pool.cc:227] Handling call
0807 11:11:04.821671 (+ 40969us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.832211 (+ 10540us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.832471 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18176}
I20260807 11:11:04.808089 25932 raft_consensus.cc:2454] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 8.
I20260807 11:11:04.783092 29970 leader_election.cc:288] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:04.789304 30112 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:04.842971 25750 leader_election.cc:302] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:04.838599 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1814) took 2315 ms. Trace:
I20260807 11:11:04.853746 26039 rpcz_store.cc:276] 0807 11:11:02.523409 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.774902 (+2251493us) service_pool.cc:227] Handling call
0807 11:11:04.821740 (+ 46838us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.825095 (+ 3355us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.838517 (+ 13422us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13696}
I20260807 11:11:04.854801 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:04.855403 26039 raft_consensus.cc:2454] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:11:04.855649 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:11:04.849715 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 452431us
I20260807 11:11:04.838348 30102 raft_consensus.cc:482] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:04.828081 26033 raft_consensus.cc:2454] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:11:04.820089 29966 raft_consensus.cc:2781] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 11:11:04.861285 26042 raft_consensus.cc:1226] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:04.866080 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1896) took 1678 ms. Trace:
I20260807 11:11:04.813922 30218 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:04.844723 26038 raft_consensus.cc:1261] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260807 11:11:04.871950 26038 pending_rounds.cc:76] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9: Aborting all ops after (but not including) 5
I20260807 11:11:04.872292 26038 pending_rounds.cc:97] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260807 11:11:04.872609 26038 raft_consensus.cc:2866] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:11:04.873536 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 498) took 2338 ms. Trace:
I20260807 11:11:04.873941 26038 rpcz_store.cc:276] 0807 11:11:02.534891 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.776330 (+2241439us) service_pool.cc:227] Handling call
0807 11:11:04.776640 (+ 310us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.873218 (+ 96578us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.873468 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:04.819162 30105 raft_consensus.cc:729] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:04.851575 25786 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:11:04.859848 25917 raft_consensus.cc:1226] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 7. Current term is 8. Ops: []
I20260807 11:11:04.821436 29765 raft_consensus.cc:2726] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:04.866447 26042 rpcz_store.cc:276] 0807 11:11:03.187343 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.826348 (+1639005us) service_pool.cc:227] Handling call
0807 11:11:04.852583 (+ 26235us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.865801 (+ 13218us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.866023 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:04.863121 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:04.880734 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:04.860438 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:04.868105 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:04.838042 30296 raft_consensus.cc:2726] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:11:04.859915 30320 raft_consensus.cc:482] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:04.813271 30288 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:04.836580 30094 raft_consensus.cc:3037] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:11:04.884717 26936 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:04.862370 30153 raft_consensus.cc:2726] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 11:11:04.899597 25801 raft_consensus.cc:1261] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 11:11:04.899883 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:04.859654 30030 raft_consensus.cc:2781] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader election won for term 7
I20260807 11:11:04.885607 26035 raft_consensus.cc:2454] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 7.
I20260807 11:11:04.882566 25926 raft_consensus.cc:1226] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: []
I20260807 11:11:04.894779 26938 leader_election.cc:302] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:04.887336 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:04.852725 29323 raft_consensus.cc:2781] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 11:11:04.885553 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:04.902858 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:04.905447 25752 leader_election.cc:302] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:04.903399 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:04.853240 29857 consensus_queue.cc:1035] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.042s
I20260807 11:11:04.883287 29966 raft_consensus.cc:686] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 7 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:04.911803 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:04.909021 25786 raft_consensus.cc:2454] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:11:04.909789 26042 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:11:04.916730 26034 raft_consensus.cc:2454] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 8.
I20260807 11:11:04.918398 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:04.919267 25931 raft_consensus.cc:2454] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:11:04.911942 26938 leader_election.cc:302] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:11:04.924588 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:04.877336 30112 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:04.859289 30102 raft_consensus.cc:504] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:04.930886 25796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 393) took 2790 ms. Trace:
I20260807 11:11:04.900592 29765 raft_consensus.cc:482] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:04.932852 25796 rpcz_store.cc:276] 0807 11:11:02.140847 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.813449 (+2672602us) service_pool.cc:227] Handling call
0807 11:11:04.837033 (+ 23584us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.837224 (+ 191us) raft_consensus.cc:1520] Early marking committed up to index 8
0807 11:11:04.865897 (+ 28673us) log.cc:842] Serialized 14 byte log entry
0807 11:11:04.930546 (+ 64649us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:04.930603 (+ 57us) raft_consensus.cc:1631] Marking committed up to 8
0807 11:11:04.930648 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.930711 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:04.930825 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27658,"prepare.queue_time_us":33152,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":28751,"raft.run_cpu_time_us":51,"raft.run_wall_time_us":51,"thread_start_us":36596,"threads_started":1}
I20260807 11:11:04.934417 26034 raft_consensus.cc:1226] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 5. Current term is 6. Ops: []
I20260807 11:11:04.930406 25752 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:04.938866 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:04.939087 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:11:04.940277 25750 connection.cc:232] RPC call timeout handler was delayed by -1.0088279321789742s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15100000000000002s for a response
I20260807 11:11:04.903508 30154 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:04.939836 25917 raft_consensus.cc:2454] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:11:04.908587 29692 raft_consensus.cc:2781] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:04.956874 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:04.907402 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:04.941570 30153 raft_consensus.cc:2781] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 11:11:04.939957 26035 raft_consensus.cc:2454] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
W20260807 11:11:04.969296 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 58061us
I20260807 11:11:04.917328 30030 raft_consensus.cc:686] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 7 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:04.954416 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:04.913569 29323 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:04.911698 30123 consensus_queue.cc:1046] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.055s
I20260807 11:11:04.912375 30169 consensus_queue.cc:1035] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.051s
I20260807 11:11:04.897775 30320 raft_consensus.cc:504] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:04.932206 30105 consensus_queue.cc:260] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:04.976506 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:05.408154 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:05.371388 25808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 392) took 3249 ms. Trace:
I20260807 11:11:05.409240 26986 raft_consensus.cc:3037] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:05.409631 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:05.409487 25808 rpcz_store.cc:276] 0807 11:11:02.121874 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.781689 (+2659815us) service_pool.cc:227] Handling call
0807 11:11:04.782032 (+ 343us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:04.782446 (+ 414us) raft_consensus.cc:1520] Early marking committed up to index 6
0807 11:11:04.782731 (+ 285us) log.cc:842] Serialized 14 byte log entry
0807 11:11:04.929975 (+147244us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:04.932219 (+ 2244us) log.cc:842] Serialized 26 byte log entry
0807 11:11:04.932396 (+ 177us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:04.932429 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.932524 (+ 95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:05.371074 (+438550us) raft_consensus.cc:1671] finished
0807 11:11:05.371104 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:05.371300 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30274,"prepare.queue_time_us":21500,"prepare.run_wall_time_us":4987,"raft.queue_time_us":62929,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":7620,"spinlock_wait_cycles":7998720,"thread_start_us":116554,"threads_started":1,"wal-append.queue_time_us":510222}
I20260807 11:11:05.069154 25801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 395) took 2573 ms. Trace:
I20260807 11:11:05.070202 25800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1810) took 3111 ms. Trace:
I20260807 11:11:04.979662 26035 raft_consensus.cc:1261] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 11:11:05.432122 26035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1904) took 1088 ms. Trace:
I20260807 11:11:05.432720 26035 rpcz_store.cc:276] 0807 11:11:04.343939 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.965562 (+621623us) service_pool.cc:227] Handling call
0807 11:11:04.979255 (+ 13693us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:05.431771 (+452516us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:05.432051 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:05.140118 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:04.976667 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:11:05.066191 25798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59118 (request call id 394) took 2647 ms (client timeout 1682 ms). Trace:
W20260807 11:11:05.035878 25925 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1778) took 2509 ms (client timeout 1653 ms). Trace:
W20260807 11:11:05.440322 25925 rpcz_store.cc:269] 0807 11:11:02.526192 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:02.529650 (+ 3458us) service_pool.cc:227] Handling call
0807 11:11:05.035583 (+2505933us) spinlock_profiling.cc:243] Waited 752 ms on lock 0x7b5400407754. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:05.035770 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1728687488}
I20260807 11:11:05.064963 25916 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1790) took 2334 ms. Trace:
I20260807 11:11:05.443940 25916 rpcz_store.cc:276] 0807 11:11:02.730991 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:02.731231 (+ 240us) service_pool.cc:227] Handling call
0807 11:11:02.817237 (+ 86006us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.080081 (+1262844us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:05.064633 (+984552us) spinlock_profiling.cc:243] Waited 576 ms on lock 0x7b5400407c54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:05.064884 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1325148544}
I20260807 11:11:05.037038 25920 raft_consensus.cc:1261] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 11:11:04.924957 30048 raft_consensus.cc:2781] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:05.196897 25926 raft_consensus.cc:1261] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:11:05.428269 25801 rpcz_store.cc:276] 0807 11:11:02.496138 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.818791 (+2322653us) service_pool.cc:227] Handling call
0807 11:11:04.847518 (+ 28727us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.899977 (+ 52459us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:05.068891 (+168914us) spinlock_profiling.cc:243] Waited 50.4 ms on lock 0x7b54003acfd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:05.069087 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":115966848}
I20260807 11:11:05.462632 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:11:05.376417 29802 consensus_meta.cc:287] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.687s user 0.011s sys 0.000s
W20260807 11:11:05.363032 30353 log.cc:924] Time spent T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.076s user 0.000s sys 0.001s
I20260807 11:11:05.061785 25918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 336) took 1728 ms. Trace:
I20260807 11:11:05.526499 25918 rpcz_store.cc:276] 0807 11:11:03.332884 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.147207 (+814323us) service_pool.cc:227] Handling call
0807 11:11:04.186910 (+ 39703us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.645210 (+458300us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:04.645262 (+ 52us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:04.645311 (+ 49us) raft_consensus.cc:1631] Marking committed up to 8
0807 11:11:04.645348 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:05.061443 (+416095us) spinlock_profiling.cc:243] Waited 458 ms on lock 0x7b540019ffd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:05.061506 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:05.061701 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1053414272}
W20260807 11:11:05.415308 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 503406us
I20260807 11:11:05.417080 30152 raft_consensus.cc:3032] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Stepping down as leader of term 6
W20260807 11:11:05.420965 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 466993us
I20260807 11:11:04.987810 26033 raft_consensus.cc:2379] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 7: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
W20260807 11:11:05.046241 30253 log.cc:924] Time spent T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 2.171s user 0.007s sys 0.015s
W20260807 11:11:05.363556 30355 log.cc:924] Time spent T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.307s user 0.001s sys 0.000s
I20260807 11:11:04.953452 28964 raft_consensus.cc:2781] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Leader election won for term 8
W20260807 11:11:05.439072 25798 rpcz_store.cc:269] 0807 11:11:02.418332 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:02.544957 (+126625us) service_pool.cc:227] Handling call
0807 11:11:05.065940 (+2520983us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b54002f6a54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:05.066103 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3387192192}
I20260807 11:11:05.438107 25931 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:11:05.042150 25919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 332) took 2842 ms. Trace:
I20260807 11:11:05.410983 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:05.513319 25925 raft_consensus.cc:1261] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260807 11:11:04.986663 30344 consensus_queue.cc:1035] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.599s
I20260807 11:11:04.960979 29692 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:05.398654 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:05.430363 25800 rpcz_store.cc:276] 0807 11:11:01.959267 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:01.972580 (+ 13313us) service_pool.cc:227] Handling call
0807 11:11:01.982434 (+ 9854us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:02.882988 (+900554us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 11:11:02.978490 (+ 95502us) spinlock_profiling.cc:243] Waited 7.34 ms on lock 0x7b48001326d8. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc456b7f1 00007f3fc45628e7 00007f3fc456277a 00007f3fc465695f 00007f3fc4656627 00007f3fc466ab2c 00007f3fc466aad8 00007f3fc466aa58 00007f3fc466aa13 00007f3fc4669cbc 00007f3fc35ebbe6
0807 11:11:03.024302 (+ 45812us) log.cc:842] Serialized 14 byte log entry
0807 11:11:04.065828 (+1041526us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:04.065907 (+ 79us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:04.065986 (+ 79us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:05.069940 (+1003954us) spinlock_profiling.cc:243] Waited 524 ms on lock 0x7b540014dcd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:05.070004 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:05.070139 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":44044,"prepare.queue_time_us":170325,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":60,"raft.queue_time_us":39926,"raft.run_cpu_time_us":1358,"raft.run_wall_time_us":535179,"spinlock_wait_cycles":1222899200,"thread_start_us":1292559,"threads_started":2,"wal-append.queue_time_us":1322478}
I20260807 11:11:05.564523 26987 raft_consensus.cc:1261] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260807 11:11:05.232311 30342 log.cc:924] Time spent T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.336s user 0.000s sys 0.000s
I20260807 11:11:05.576972 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:11:05.207940 29851 consensus_meta.cc:287] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.509s user 0.005s sys 0.006s
W20260807 11:11:05.064157 30235 log.cc:924] Time spent T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 3.011s user 0.018s sys 0.006s
W20260807 11:11:05.292019 30354 log.cc:924] Time spent T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.283s user 0.000s sys 0.000s
I20260807 11:11:04.962532 30153 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:05.414291 30288 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:11:05.290995 30348 log.cc:924] Time spent T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.387s user 0.000s sys 0.001s
I20260807 11:11:05.004745 29977 consensus_queue.cc:1035] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.674s
I20260807 11:11:05.013093 29857 consensus_queue.cc:1046] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.702s
I20260807 11:11:04.961333 30154 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:05.006683 30296 consensus_queue.cc:1035] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.710s
I20260807 11:11:05.291250 30309 consensus_queue.cc:1035] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.852s
I20260807 11:11:05.319036 30030 consensus_queue.cc:237] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:11:05.938494 30218 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.962s user 0.008s sys 0.008s
I20260807 11:11:05.238518 29966 consensus_queue.cc:237] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:05.068246 26974 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50942 (request call id 473) took 2639 ms. Trace:
I20260807 11:11:05.610126 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:05.189601 30298 raft_consensus.cc:482] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:05.760155 30369 raft_consensus.cc:2741] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260807 11:11:06.408896 30298 raft_consensus.cc:504] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:06.379429 30094 consensus_meta.cc:287] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 1.480s user 0.014s sys 0.000s
I20260807 11:11:04.973732 29765 raft_consensus.cc:504] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:06.411031 29644 raft_consensus.cc:482] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:06.411561 29644 raft_consensus.cc:504] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:06.412009 30298 leader_election.cc:288] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:06.412734 29765 leader_election.cc:288] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:11:06.361614 30370 log.cc:924] Time spent T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.484s user 0.000s sys 0.000s
I20260807 11:11:06.414440 25808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1841) took 1557 ms. Trace:
I20260807 11:11:05.616251 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:11:05.811767 26939 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.697s user 0.007s sys 0.028s
W20260807 11:11:06.415573 26939 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.301s user 0.015s sys 0.031s
I20260807 11:11:05.472218 29323 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Advancing to term 9
I20260807 11:11:06.416049 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1897) took 2322 ms. Trace:
I20260807 11:11:06.416218 25796 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:11:06.416817 25796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59118 (request call id 404) took 1565 ms. Trace:
I20260807 11:11:05.039758 29680 raft_consensus.cc:3032] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:11:06.417681 29680 raft_consensus.cc:729] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:06.416570 26039 rpcz_store.cc:276] 0807 11:11:04.093392 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.856013 (+762621us) service_pool.cc:227] Handling call
0807 11:11:04.856325 (+ 312us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:04.870639 (+ 14314us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:04.870671 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:04.882269 (+ 11598us) log.cc:842] Serialized 26 byte log entry
0807 11:11:04.998892 (+116623us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:11:04.999249 (+ 357us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.999659 (+ 410us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:06.415847 (+1416188us) raft_consensus.cc:1671] finished
0807 11:11:06.415867 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:06.415988 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":91,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":16768,"thread_start_us":84972,"threads_started":1,"wal-append.queue_time_us":785791}
I20260807 11:11:06.418883 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:11:05.680154 26042 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.682s user 0.009s sys 0.005s
W20260807 11:11:06.420186 26042 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.422s user 0.013s sys 0.009s
I20260807 11:11:06.415025 25931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35276 (request call id 343) took 1554 ms. Trace:
I20260807 11:11:06.421054 25931 rpcz_store.cc:276] 0807 11:11:04.861024 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.930926 (+ 69902us) service_pool.cc:227] Handling call
0807 11:11:06.414968 (+1484042us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:05.656153 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:05.069353 30329 raft_consensus.cc:2726] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:11:06.429534 26987 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50942 (request call id 474) took 3735 ms. Trace:
I20260807 11:11:06.415838 26980 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 485) took 1526 ms. Trace:
W20260807 11:11:05.936280 26038 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.005s user 0.064s sys 0.133s
W20260807 11:11:05.905702 26037 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.862s user 0.017s sys 0.078s
W20260807 11:11:05.715618 29417 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.775s user 0.004s sys 0.011s
W20260807 11:11:06.129204 25752 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.132s user 0.002s sys 0.010s
I20260807 11:11:06.413772 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:11:05.290963 30356 log.cc:924] Time spent T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.277s user 0.000s sys 0.000s
I20260807 11:11:05.312227 30334 leader_election.cc:288] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:06.439378 26985 raft_consensus.cc:2362] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 7. Re-sending same reply.
I20260807 11:11:05.476504 30048 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 11:11:05.937211 26937 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.911s user 0.007s sys 0.009s
W20260807 11:11:06.443287 26937 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.417s user 0.007s sys 0.024s
I20260807 11:11:06.413723 25925 pending_rounds.cc:76] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc: Aborting all ops after (but not including) 2
W20260807 11:11:05.747395 25750 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.792s user 0.000s sys 0.017s
I20260807 11:11:06.444574 25925 pending_rounds.cc:97] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc: Aborting uncommitted NO_OP operation due to leader change: 4.3
W20260807 11:11:06.444706 25750 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.490s user 0.010s sys 0.038s
I20260807 11:11:06.444947 25925 raft_consensus.cc:2866] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260807 11:11:06.445145 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 1.49025e+06us
I20260807 11:11:06.445643 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1806) took 1563 ms. Trace:
I20260807 11:11:06.446008 25750 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:06.413442 25916 raft_consensus.cc:1226] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 5. Current term is 6. Ops: []
I20260807 11:11:06.177688 25920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1797) took 1319 ms. Trace:
I20260807 11:11:06.447046 25920 rpcz_store.cc:276] 0807 11:11:04.858527 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.858657 (+ 130us) service_pool.cc:227] Handling call
0807 11:11:05.036531 (+177874us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:05.445907 (+409376us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:06.177395 (+731488us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b5400407750. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:06.177597 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":365699456}
W20260807 11:11:06.208860 25786 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59118 (request call id 405) took 1308 ms (client timeout 1696 ms). Trace:
I20260807 11:11:06.447908 25750 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:11:06.448100 25786 rpcz_store.cc:269] 0807 11:11:04.870830 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.886989 (+ 16159us) service_pool.cc:227] Handling call
0807 11:11:06.178932 (+1291943us) spinlock_profiling.cc:243] Waited 3.81 ms on lock 0x7b540014e6d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:06.179066 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8758528}
W20260807 11:11:06.424700 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 676278us
I20260807 11:11:06.415006 25808 rpcz_store.cc:276] 0807 11:11:04.857273 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:05.423647 (+566374us) service_pool.cc:227] Handling call
0807 11:11:06.414378 (+990731us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:11:06.425581 26938 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.763s user 0.004s sys 0.007s
W20260807 11:11:06.450366 26938 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.834s user 0.005s sys 0.007s
I20260807 11:11:06.426720 26035 raft_consensus.cc:3037] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
W20260807 11:11:06.406790 30218 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.444s user 0.011s sys 0.019s
I20260807 11:11:06.420508 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:11:05.960032 25869 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.101s user 0.005s sys 0.013s
I20260807 11:11:06.408386 25801 raft_consensus.cc:2396] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260807 11:11:06.455780 25869 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.597s user 0.012s sys 0.020s
W20260807 11:11:06.456164 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 1.5976e+06us
I20260807 11:11:06.429976 26987 rpcz_store.cc:276] 0807 11:11:02.694447 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:02.694749 (+ 302us) service_pool.cc:227] Handling call
0807 11:11:05.525033 (+2830284us) spinlock_profiling.cc:243] Waited 65.5 ms on lock 0x7b5800000f98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc461a2b0 00007f3fc46057f9 00007f3fc46052a2 00007f3fbdceb684
0807 11:11:05.525337 (+ 304us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:06.421379 (+896042us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:06.429473 (+ 8094us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":150700288}
W20260807 11:11:06.266620 25755 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.005s sys 0.026s
I20260807 11:11:06.406659 28964 raft_consensus.cc:686] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 8 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:05.246850 30108 consensus_queue.cc:1046] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 1.202s
W20260807 11:11:06.438848 29417 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.510s user 0.007s sys 0.015s
W20260807 11:11:06.433590 26037 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.390s user 0.018s sys 0.083s
I20260807 11:11:06.419368 25919 rpcz_store.cc:276] 0807 11:11:02.199520 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:02.199665 (+ 145us) service_pool.cc:227] Handling call
0807 11:11:02.199970 (+ 305us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.170693 (+1970723us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:05.041902 (+871209us) spinlock_profiling.cc:243] Waited 1.97 s on lock 0x7b54001df954. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:05.042091 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4530452352}
W20260807 11:11:06.433436 26038 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.502s user 0.077s sys 0.134s
I20260807 11:11:06.434118 26980 rpcz_store.cc:276] 0807 11:11:04.889896 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:05.437658 (+547762us) service_pool.cc:227] Handling call
0807 11:11:06.415779 (+978121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:06.417193 25796 rpcz_store.cc:276] 0807 11:11:04.851766 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:05.410735 (+558969us) service_pool.cc:227] Handling call
0807 11:11:06.416744 (+1006009us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:11:05.599797 30184 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.011s sys 0.020s
I20260807 11:11:06.437851 26036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1850) took 7707 ms. Trace:
W20260807 11:11:06.327741 30355 log.cc:924] Time spent T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.118s user 0.000s sys 0.000s
W20260807 11:11:05.871595 30216 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.917s user 0.000s sys 0.026s
W20260807 11:11:06.141865 30363 log.cc:924] Time spent T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.392s user 0.001s sys 0.000s
W20260807 11:11:06.440548 26985 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 467) took 4949 ms (client timeout 3203 ms). Trace:
W20260807 11:11:06.482988 26985 rpcz_store.cc:269] 0807 11:11:01.491629 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:01.493431 (+ 1802us) service_pool.cc:227] Handling call
0807 11:11:06.413613 (+4920182us) spinlock_profiling.cc:243] Waited 1.57 ms on lock 0x7b5800000f98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc46061b0 00007f3fc4605a94 00007f3fbdcedc04 00007f3fbdcedb92
0807 11:11:06.440475 (+ 26862us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3612416}
W20260807 11:11:06.439105 25752 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.442s user 0.005s sys 0.013s
W20260807 11:11:06.484740 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 1.48796e+06us
I20260807 11:11:05.938441 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 493) took 3938 ms. Trace:
I20260807 11:11:05.257215 30343 consensus_queue.cc:1035] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 1.572s
I20260807 11:11:06.496439 26980 raft_consensus.cc:1226] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: []
W20260807 11:11:06.471501 30184 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.417s user 0.018s sys 0.031s
I20260807 11:11:06.516844 26980 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 496) took 1622 ms. Trace:
I20260807 11:11:06.517297 26980 rpcz_store.cc:276] 0807 11:11:04.886777 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.469439 (+1582662us) service_pool.cc:227] Handling call
0807 11:11:06.496245 (+ 26806us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:06.508386 (+ 12141us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:06.508617 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:06.517709 30288 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:06.518229 30288 raft_consensus.cc:504] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:06.518589 30218 consensus_queue.cc:1035] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 11:11:06.520340 30288 leader_election.cc:288] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:06.407516 26974 rpcz_store.cc:276] 0807 11:11:02.428522 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:02.497782 (+ 69260us) service_pool.cc:227] Handling call
0807 11:11:02.549037 (+ 51255us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:03.120627 (+571590us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:03.120670 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:03.121243 (+ 573us) log.cc:842] Serialized 26 byte log entry
0807 11:11:05.067478 (+1946235us) spinlock_profiling.cc:243] Waited 2.59 ms on lock 0x7b540068cca0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649af0 00007f3fbf654028 00007f3fbf6c4a7b 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c
0807 11:11:05.067564 (+ 86us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:05.067639 (+ 75us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:05.067766 (+ 127us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:05.067829 (+ 63us) raft_consensus.cc:1671] finished
0807 11:11:05.067851 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:05.068171 (+ 320us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":14554,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":18032,"spinlock_wait_cycles":5946240,"thread_start_us":1162369,"threads_started":1,"wal-append.queue_time_us":1175610,"wal-append.run_cpu_time_us":8715,"wal-append.run_wall_time_us":1111083}
I20260807 11:11:06.508961 29680 consensus_queue.cc:260] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:06.523183 29680 raft_consensus.cc:3037] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:11:06.531028 28964 consensus_queue.cc:237] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:06.477842 30204 raft_consensus.cc:482] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:06.533428 30204 raft_consensus.cc:504] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:06.535682 30204 leader_election.cc:288] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:06.538249 29323 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:06.543056 26979 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50942 (request call id 477) took 3267 ms. Trace:
W20260807 11:11:06.477190 30216 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.536s user 0.006s sys 0.040s
W20260807 11:11:06.550516 30241 log.cc:924] Time spent T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 4.035s user 0.022s sys 0.034s
I20260807 11:11:06.465255 30298 consensus_queue.cc:1035] T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.074s
W20260807 11:11:06.442377 26936 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.010s user 0.009s sys 0.016s
I20260807 11:11:06.445981 25925 rpcz_store.cc:276] 0807 11:11:04.882264 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:05.441913 (+559649us) service_pool.cc:227] Handling call
0807 11:11:05.442332 (+ 419us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:06.445410 (+1003078us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:06.445594 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:06.449558 25916 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1807) took 1566 ms. Trace:
W20260807 11:11:06.555629 26936 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.124s user 0.009s sys 0.016s
W20260807 11:11:06.556077 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 1.12423e+06us
I20260807 11:11:06.555949 25916 rpcz_store.cc:276] 0807 11:11:04.883054 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:05.445313 (+562259us) service_pool.cc:227] Handling call
0807 11:11:05.606530 (+161217us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:06.449347 (+842817us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:06.449509 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:06.556200 30374 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:06.557055 26936 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:11:05.597893 30105 raft_consensus.cc:3037] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
W20260807 11:11:06.128939 25727 thread.cc:636] client-negotiator [worker] (thread pool) Time spent creating pthread: real 1.142s user 0.025s sys 0.031s
I20260807 11:11:05.564916 29991 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:06.084339 30320 leader_election.cc:288] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:06.415407 30309 consensus_queue.cc:1225] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer 4fd6381544574ddca559ce63b9243efc log is divergent from this leader: its last log entry 6.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260807 11:11:06.414691 29857 consensus_queue.cc:1225] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer 4fd6381544574ddca559ce63b9243efc log is divergent from this leader: its last log entry 5.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260807 11:11:06.452805 25750 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:05.597538 30152 raft_consensus.cc:729] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:06.563596 30378 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
W20260807 11:11:06.458227 25755 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.480s user 0.011s sys 0.028s
I20260807 11:11:06.413878 29692 raft_consensus.cc:3037] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:06.495129 26041 rpcz_store.cc:276] 0807 11:11:02.000457 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.705680 (+2705223us) service_pool.cc:227] Handling call
0807 11:11:04.705999 (+ 319us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:04.706362 (+ 363us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:04.706385 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:04.706824 (+ 439us) log.cc:842] Serialized 26 byte log entry
0807 11:11:04.820739 (+113915us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:04.974432 (+153693us) log.cc:842] Serialized 14 byte log entry
0807 11:11:04.975203 (+ 771us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.975291 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:05.938136 (+962845us) raft_consensus.cc:1671] finished
0807 11:11:05.938160 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:05.938366 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":427802,"prepare.queue_time_us":743424,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":4197,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":16078,"thread_start_us":229165,"threads_started":2,"wal-append.queue_time_us":181290}
I20260807 11:11:06.557154 30374 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:06.451753 30126 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:06.610785 30126 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:05.276361 30351 consensus_queue.cc:1035] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 1.563s
I20260807 11:11:06.460575 29851 raft_consensus.cc:2726] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:05.674244 25926 raft_consensus.cc:1261] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 11:11:06.614166 25926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1810) took 1704 ms. Trace:
I20260807 11:11:06.614503 25926 rpcz_store.cc:276] 0807 11:11:04.909454 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:05.483776 (+574322us) service_pool.cc:227] Handling call
0807 11:11:05.672703 (+188927us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:06.613876 (+941173us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:06.614108 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:06.428519 30153 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:06.462015 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59312 (request call id 1824) took 1008 ms. Trace:
I20260807 11:11:06.619652 26039 rpcz_store.cc:276] 0807 11:11:05.453352 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.420281 (+966929us) service_pool.cc:227] Handling call
0807 11:11:06.461936 (+ 41655us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:06.462589 25801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1842) took 1600 ms. Trace:
I20260807 11:11:06.621289 25801 rpcz_store.cc:276] 0807 11:11:04.862688 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:05.609779 (+747091us) service_pool.cc:227] Handling call
0807 11:11:06.462510 (+852731us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:06.622584 25801 raft_consensus.cc:1261] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20260807 11:11:06.622890 25801 pending_rounds.cc:76] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815: Aborting all ops after (but not including) 8
I20260807 11:11:06.623109 25801 pending_rounds.cc:97] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20260807 11:11:06.623344 25801 raft_consensus.cc:2866] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:11:06.432013 30154 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:06.568115 30216 raft_consensus.cc:2781] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 11:11:06.471740 26036 rpcz_store.cc:276] 0807 11:10:58.721068 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:58.732926 (+ 11858us) service_pool.cc:227] Handling call
0807 11:11:01.115965 (+2383039us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:01.136832 (+ 20867us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:06.428373 (+5291541us) spinlock_profiling.cc:243] Waited 2.38 s on lock 0x7b54000d0550. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:06.428532 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5479387520}
W20260807 11:11:06.472417 25872 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.005s sys 0.018s
W20260807 11:11:06.642937 25872 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.766s user 0.006s sys 0.018s
I20260807 11:11:06.472311 25981 leader_election.cc:302] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
W20260807 11:11:06.473068 26034 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.051s user 0.009s sys 0.025s
I20260807 11:11:06.475028 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1782) took 7112 ms. Trace:
W20260807 11:11:06.645792 26034 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.224s user 0.009s sys 0.025s
I20260807 11:11:06.452580 30048 raft_consensus.cc:3037] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:05.599814 29802 raft_consensus.cc:504] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:06.479250 26987 raft_consensus.cc:1261] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 11:11:06.651330 26987 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 495) took 1767 ms. Trace:
I20260807 11:11:06.651779 26987 rpcz_store.cc:276] 0807 11:11:04.883928 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.464967 (+1581039us) service_pool.cc:227] Handling call
0807 11:11:06.478630 (+ 13663us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:06.651009 (+172379us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:06.651264 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:06.461295 29765 raft_consensus.cc:2726] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:11:06.543526 26979 rpcz_store.cc:276] 0807 11:11:03.275480 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.403261 (+1127781us) service_pool.cc:227] Handling call
0807 11:11:04.414615 (+ 11354us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:04.415023 (+ 408us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:04.415052 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:04.415521 (+ 469us) log.cc:842] Serialized 26 byte log entry
0807 11:11:04.565403 (+149882us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:04.565443 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.565524 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:06.542790 (+1977266us) raft_consensus.cc:1671] finished
0807 11:11:06.542817 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:06.542984 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1109,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":106,"thread_start_us":149489,"threads_started":1,"wal-append.queue_time_us":412819}
I20260807 11:11:06.558876 26936 leader_election.cc:302] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
W20260807 11:11:06.560859 25727 thread.cc:602] client-negotiator [worker] (thread pool) Time spent starting thread: real 1.574s user 0.030s sys 0.032s
W20260807 11:11:06.441672 25917 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.590s user 0.009s sys 0.006s
I20260807 11:11:06.569272 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 496) took 4364 ms. Trace:
I20260807 11:11:06.559170 30296 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
W20260807 11:11:06.426434 26986 consensus_meta.cc:287] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 1.017s user 0.010s sys 0.001s
I20260807 11:11:06.671526 26986 raft_consensus.cc:2454] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 8.
W20260807 11:11:06.671861 25917 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.820s user 0.009s sys 0.006s
W20260807 11:11:06.459218 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 929815us
I20260807 11:11:06.595757 30108 raft_consensus.cc:2726] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:11:06.571859 30391 raft_consensus.cc:482] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:06.592813 30152 consensus_queue.cc:260] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:06.624516 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:06.469933 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1898) took 2376 ms. Trace:
I20260807 11:11:06.697938 25786 raft_consensus.cc:1261] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260807 11:11:06.698256 26038 rpcz_store.cc:276] 0807 11:11:04.093745 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.874885 (+781140us) service_pool.cc:227] Handling call
0807 11:11:04.875276 (+ 391us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.875527 (+ 251us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 11:11:04.931397 (+ 55870us) log.cc:842] Serialized 14 byte log entry
0807 11:11:06.466276 (+1534879us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:06.469633 (+ 3357us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:06.469676 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:06.469734 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:06.469858 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":907479,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"raft.queue_time_us":322,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":6347,"thread_start_us":1572001,"threads_started":2,"wal-append.queue_time_us":1647034}
I20260807 11:11:06.603547 26035 raft_consensus.cc:2454] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 7.
I20260807 11:11:06.701274 26035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59312 (request call id 1821) took 1264 ms. Trace:
I20260807 11:11:06.701917 25798 raft_consensus.cc:1226] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:06.612430 30374 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:11:06.628145 25797 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59118 (request call id 384) took 4602 ms (client timeout 2260 ms). Trace:
I20260807 11:11:06.629463 30153 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:06.631440 29323 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 9 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:06.632833 25801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 397) took 2485 ms. Trace:
I20260807 11:11:06.715018 25801 rpcz_store.cc:276] 0807 11:11:04.147490 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.621935 (+2474445us) service_pool.cc:227] Handling call
0807 11:11:06.622258 (+ 323us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:06.623790 (+ 1532us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:06.632742 (+ 8952us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:06.716980 25801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 401) took 2441 ms. Trace:
I20260807 11:11:06.635536 25803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1821) took 4672 ms. Trace:
I20260807 11:11:06.717500 30270 consensus_queue.cc:1035] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:11:06.717681 25803 rpcz_store.cc:276] 0807 11:11:01.963287 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.162814 (+2199527us) service_pool.cc:227] Handling call
0807 11:11:04.171411 (+ 8597us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.658372 (+486961us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:04.658415 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:04.658470 (+ 55us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:11:04.658517 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:06.635308 (+1976791us) spinlock_profiling.cc:243] Waited 487 ms on lock 0x7b54000177d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:06.635352 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:06.635480 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1119288704}
I20260807 11:11:06.719947 25803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1834) took 2003 ms. Trace:
I20260807 11:11:06.562664 29991 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:06.722399 29991 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:06.571929 29857 consensus_queue.cc:1035] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
I20260807 11:11:04.568804 25921 rpcz_store.cc:276] 0807 11:11:03.236010 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:03.236186 (+ 176us) service_pool.cc:227] Handling call
0807 11:11:03.254511 (+ 18325us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:03.254780 (+ 269us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 11:11:03.612625 (+357845us) log.cc:842] Serialized 14 byte log entry
0807 11:11:04.417345 (+804720us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:04.417394 (+ 49us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:04.417436 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.417504 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:04.417607 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38377,"prepare.queue_time_us":1002047,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":554300,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":40468,"thread_start_us":1063097,"threads_started":3,"wal-append.queue_time_us":786350}
W20260807 11:11:06.638691 26033 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43246 (request call id 505) took 1778 ms (client timeout 1958 ms). Trace:
I20260807 11:11:06.646793 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1901) took 2514 ms. Trace:
I20260807 11:11:06.717360 25801 rpcz_store.cc:276] 0807 11:11:04.275498 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.715920 (+2440422us) service_pool.cc:227] Handling call
0807 11:11:06.716256 (+ 336us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:06.716590 (+ 334us) raft_consensus.cc:1520] Early marking committed up to index 7
0807 11:11:06.716622 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:06.716667 (+ 45us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:06.716706 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:06.716785 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:06.716923 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:06.725127 25808 raft_consensus.cc:3032] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:11:06.725530 25808 raft_consensus.cc:729] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:06.726090 30320 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:11:06.723757 25799 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59118 (request call id 402) took 2443 ms (client timeout 1531 ms). Trace:
I20260807 11:11:06.726259 25808 consensus_queue.cc:260] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:06.726603 30320 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:06.726859 25799 rpcz_store.cc:269] 0807 11:11:04.280737 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.296296 (+ 15559us) service_pool.cc:227] Handling call
0807 11:11:06.723568 (+2427272us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b54001fed54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:06.723689 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":348034432}
I20260807 11:11:06.727451 25808 raft_consensus.cc:3037] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:11:06.728211 25799 pending_rounds.cc:76] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815: Aborting all ops after (but not including) 1
I20260807 11:11:06.728492 25799 pending_rounds.cc:97] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260807 11:11:06.728751 25799 raft_consensus.cc:2866] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260807 11:11:06.723551 26033 rpcz_store.cc:269] 0807 11:11:04.860182 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.862815 (+ 2633us) service_pool.cc:227] Handling call
0807 11:11:06.638458 (+1775643us) spinlock_profiling.cc:243] Waited 91.5 ms on lock 0x7b54000a3a54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:06.638622 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":210409728}
I20260807 11:11:06.645922 26037 rpcz_store.cc:276] 0807 11:10:59.362533 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:01.813356 (+2450823us) service_pool.cc:227] Handling call
0807 11:11:01.813694 (+ 338us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:01.813925 (+ 231us) raft_consensus.cc:1520] Early marking committed up to index 6
0807 11:11:01.821793 (+ 7868us) log.cc:842] Serialized 14 byte log entry
0807 11:11:05.043433 (+3221640us) spinlock_profiling.cc:243] Waited 7.43 ms on lock 0x7b18004086a0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbc77e345 00007f3fc55b5a39 00007f3fbc79b8bb 00007f3fbc799cbf 00007f3fbc79be22 00007f3fbf3455bb 00007f3fbf34b063 00007f3fbf34b57b 00007f3fbf6ce88a
0807 11:11:06.474730 (+1431297us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:06.474782 (+ 52us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:06.474846 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:06.474906 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:06.474984 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":3600,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":1296,"raft.queue_time_us":26761,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":28423,"spinlock_wait_cycles":17099392,"thread_start_us":1422381,"threads_started":1,"wal-append.queue_time_us":4640810}
I20260807 11:11:06.648926 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:06.649698 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:11:06.670740 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 198512us
W20260807 11:11:06.672175 26986 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 494) took 1788 ms (client timeout 2084 ms). Trace:
I20260807 11:11:06.720293 25803 rpcz_store.cc:276] 0807 11:11:04.716613 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.719079 (+2002466us) service_pool.cc:227] Handling call
0807 11:11:06.719391 (+ 312us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:06.719597 (+ 206us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:06.719625 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:06.719670 (+ 45us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:06.719710 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:06.719764 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:06.719891 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:06.732904 26036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59312 (request call id 1820) took 1297 ms. Trace:
W20260807 11:11:06.732806 26986 rpcz_store.cc:269] 0807 11:11:04.883572 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.972657 (+ 89085us) service_pool.cc:227] Handling call
0807 11:11:06.672118 (+1699461us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:06.733459 26036 rpcz_store.cc:276] 0807 11:11:05.435897 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.649377 (+1213480us) service_pool.cc:227] Handling call
0807 11:11:06.732847 (+ 83470us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:06.671342 26046 rpcz_store.cc:276] 0807 11:11:02.205107 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.714755 (+2509648us) service_pool.cc:227] Handling call
0807 11:11:04.715067 (+ 312us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:04.744250 (+ 29183us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 11:11:04.744281 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:04.804139 (+ 59858us) log.cc:842] Serialized 26 byte log entry
0807 11:11:04.893381 (+ 89242us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:04.893416 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:04.893478 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:06.569037 (+1675559us) raft_consensus.cc:1671] finished
0807 11:11:06.569057 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:06.569209 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":31960,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":114,"thread_start_us":40932,"threads_started":1,"wal-append.queue_time_us":610355}
I20260807 11:11:06.673290 26938 leader_election.cc:302] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:11:06.734619 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 61385.7us
W20260807 11:11:06.674141 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 115362us
I20260807 11:11:06.697656 26039 raft_consensus.cc:2454] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 7.
W20260807 11:11:06.705492 25797 rpcz_store.cc:269] 0807 11:11:02.025334 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:02.139313 (+113979us) service_pool.cc:227] Handling call
0807 11:11:06.627950 (+4488637us) spinlock_profiling.cc:243] Waited 2.01 s on lock 0x7b54001ff254. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:06.628077 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4612678016}
I20260807 11:11:06.736394 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59312 (request call id 1822) took 1284 ms. Trace:
I20260807 11:11:06.701786 26035 rpcz_store.cc:276] 0807 11:11:05.436240 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:05.436533 (+ 293us) service_pool.cc:227] Handling call
0807 11:11:06.701019 (+1264486us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:06.670464 29608 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:06.710851 30124 consensus_queue.cc:1046] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 11:11:06.670303 29950 raft_consensus.cc:2726] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:11:06.711637 30152 raft_consensus.cc:3037] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:06.703627 30391 raft_consensus.cc:504] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:06.712980 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:06.700048 30172 raft_consensus.cc:2781] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:11:06.761359 25981 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:06.761587 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:06.761898 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:06.762492 26037 raft_consensus.cc:2396] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 8 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260807 11:11:06.724236 26034 rpcz_store.cc:276] 0807 11:11:04.131928 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.957500 (+825572us) service_pool.cc:227] Handling call
0807 11:11:04.998459 (+ 40959us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:06.646333 (+1647874us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:06.646383 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:06.646445 (+ 62us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:06.646500 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:06.646588 (+ 88us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:06.646728 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":473099,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":76160,"thread_start_us":1224159,"threads_started":1}
I20260807 11:11:06.763120 26036 raft_consensus.cc:2396] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260807 11:11:06.723706 29802 leader_election.cc:288] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:06.736793 26936 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:06.657059 30344 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:06.737551 26035 raft_consensus.cc:3032] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 11:11:06.736720 26039 rpcz_store.cc:276] 0807 11:11:05.452516 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.620552 (+1168036us) service_pool.cc:227] Handling call
0807 11:11:06.736342 (+115790us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:06.766089 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:06.766968 26046 raft_consensus.cc:2454] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:11:06.657230 30216 raft_consensus.cc:686] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:06.735577 26938 leader_election.cc:302] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:06.769590 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1899) took 2675 ms. Trace:
W20260807 11:11:06.768918 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 304781us
I20260807 11:11:06.730322 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:06.769991 26042 rpcz_store.cc:276] 0807 11:11:04.094052 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.929558 (+835506us) service_pool.cc:227] Handling call
0807 11:11:04.959665 (+ 30107us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:04.959952 (+ 287us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 11:11:06.420688 (+1460736us) log.cc:842] Serialized 14 byte log entry
0807 11:11:06.769271 (+348583us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:06.769327 (+ 56us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:06.769376 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:06.769437 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:06.769538 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5809,"prepare.queue_time_us":934796,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":67,"thread_start_us":1764992,"threads_started":2}
I20260807 11:11:06.702984 25917 raft_consensus.cc:2932] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:11:06.670524 30298 raft_consensus.cc:482] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:06.737950 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59312 (request call id 1823) took 1285 ms. Trace:
I20260807 11:11:06.775220 26041 rpcz_store.cc:276] 0807 11:11:05.452968 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.643063 (+1190095us) service_pool.cc:227] Handling call
0807 11:11:06.737890 (+ 94827us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:06.767390 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59312 (request call id 1826) took 1313 ms. Trace:
I20260807 11:11:06.775738 26033 raft_consensus.cc:2396] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260807 11:11:06.765656 26038 raft_consensus.cc:2454] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:11:06.776010 26046 rpcz_store.cc:276] 0807 11:11:05.454173 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.734312 (+1280139us) service_pool.cc:227] Handling call
0807 11:11:06.767329 (+ 33017us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:06.709147 29765 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:06.773905 26039 raft_consensus.cc:1226] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 9 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260807 11:11:06.766157 26035 raft_consensus.cc:729] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:11:06.740274 30162 raft_consensus.cc:2781] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:06.787781 26041 raft_consensus.cc:1261] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260807 11:11:06.786978 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59312 (request call id 1825) took 1333 ms. Trace:
I20260807 11:11:06.788707 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 502) took 1997 ms. Trace:
I20260807 11:11:06.789208 26041 rpcz_store.cc:276] 0807 11:11:04.791497 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.775826 (+1984329us) service_pool.cc:227] Handling call
0807 11:11:06.787394 (+ 11568us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:06.788390 (+ 996us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:06.788631 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:06.783507 25808 raft_consensus.cc:1261] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260807 11:11:06.792405 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1910) took 2050 ms. Trace:
I20260807 11:11:06.792729 25808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1829) took 3512 ms. Trace:
I20260807 11:11:06.792923 26039 rpcz_store.cc:276] 0807 11:11:04.741941 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.766938 (+2024997us) service_pool.cc:227] Handling call
0807 11:11:06.767354 (+ 416us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:06.792054 (+ 24700us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:06.792334 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:06.788954 26033 rpcz_store.cc:276] 0807 11:11:05.453839 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.730075 (+1276236us) service_pool.cc:227] Handling call
0807 11:11:06.786898 (+ 56823us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:06.796705 25796 raft_consensus.cc:1226] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 7 LEADER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:06.793296 25808 rpcz_store.cc:276] 0807 11:11:03.280160 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.463639 (+3183479us) service_pool.cc:227] Handling call
0807 11:11:06.476047 (+ 12408us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:06.725035 (+248988us) spinlock_profiling.cc:243] Waited 87.6 ms on lock 0x7b54001e5b20. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf654730 00007f3fbf6c6300 00007f3fbf6c7371 00007f3fbf6c3edf 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc
0807 11:11:06.792422 (+ 67387us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:06.792666 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":201428224}
I20260807 11:11:06.728622 30320 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:06.809439 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:06.810217 26041 raft_consensus.cc:1261] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 9 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260807 11:11:06.810839 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 503) took 1992 ms. Trace:
I20260807 11:11:06.811367 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1907) took 2423 ms. Trace:
I20260807 11:11:06.812134 26034 rpcz_store.cc:276] 0807 11:11:04.388460 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.763389 (+2374929us) service_pool.cc:227] Handling call
0807 11:11:06.810770 (+ 47381us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:06.810978 (+ 208us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:06.811016 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:06.811072 (+ 56us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:06.811122 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:06.811176 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:06.811295 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:06.811218 26041 rpcz_store.cc:276] 0807 11:11:04.819076 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.790122 (+1971046us) service_pool.cc:227] Handling call
0807 11:11:06.809903 (+ 19781us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:06.810592 (+ 689us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:06.810781 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:06.761240 30172 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:06.815606 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 500) took 2024 ms. Trace:
I20260807 11:11:06.862870 26037 rpcz_store.cc:276] 0807 11:11:04.790857 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.774903 (+1984046us) service_pool.cc:227] Handling call
0807 11:11:06.815102 (+ 40199us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:06.815305 (+ 203us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:06.815328 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:06.815368 (+ 40us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:06.815401 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:06.815450 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:06.815546 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:11:06.784752 30210 log.cc:924] Time spent T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 4.862s user 0.016s sys 0.030s
W20260807 11:11:06.836848 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 47374.2us
I20260807 11:11:06.837095 30124 raft_consensus.cc:3032] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 5 LEADER]: Stepping down as leader of term 5
I20260807 11:11:06.849319 29966 raft_consensus.cc:482] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:06.859915 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:06.877485 26034 raft_consensus.cc:1261] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260807 11:11:06.816133 26036 raft_consensus.cc:1261] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 11:11:06.706436 30296 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:06.768215 29323 raft_consensus.cc:2781] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:11:06.642690 30378 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:06.871279 25869 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:06.877676 26046 raft_consensus.cc:2454] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:11:06.873713 25797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:53306 (request call id 704) took 2584 ms. Trace:
I20260807 11:11:06.861385 26035 consensus_queue.cc:260] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:06.878098 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1816) took 1737 ms. Trace:
I20260807 11:11:06.809744 30162 raft_consensus.cc:482] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 11:11:06.860859 30266 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260807 11:11:06.754943 29608 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:06.784353 29950 raft_consensus.cc:2726] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:11:06.895774 25797 rpcz_store.cc:276] 0807 11:11:04.266620 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.736501 (+2469881us) service_pool.cc:227] Handling call
0807 11:11:06.809458 (+ 72957us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:06.897030 26938 leader_election.cc:302] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:06.896039 26035 raft_consensus.cc:3037] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:11:06.768064 30219 consensus_queue.cc:1046] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.107s
I20260807 11:11:06.785442 30398 raft_consensus.cc:2726] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260807 11:11:06.899300 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 339500us
I20260807 11:11:06.870616 30172 raft_consensus.cc:3037] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Advancing to term 6
W20260807 11:11:06.878304 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 425666us
I20260807 11:11:06.878440 26036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1911) took 2136 ms. Trace:
I20260807 11:11:06.881347 25981 leader_election.cc:302] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:11:06.881898 26042 raft_consensus.cc:3037] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:06.708484 30404 consensus_queue.cc:1046] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.173s
I20260807 11:11:06.868888 30397 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:11:06.916091 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 223969us
I20260807 11:11:06.896469 26034 rpcz_store.cc:276] 0807 11:11:05.140831 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.813450 (+1672619us) service_pool.cc:227] Handling call
0807 11:11:06.877177 (+ 63727us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:06.877860 (+ 683us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:06.878045 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:06.868515 29680 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:06.774523 30344 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:06.899215 29323 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:06.910282 30162 raft_consensus.cc:3037] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:06.923754 26046 raft_consensus.cc:1226] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 4. Current term is 5. Ops: []
W20260807 11:11:06.950414 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 53487.8us
I20260807 11:11:06.951344 26046 raft_consensus.cc:1226] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:06.922127 26036 rpcz_store.cc:276] 0807 11:11:04.742260 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.774876 (+2032616us) service_pool.cc:227] Handling call
0807 11:11:06.815805 (+ 40929us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:06.878195 (+ 62390us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:06.878388 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:11:06.951620 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 86286.3us
I20260807 11:11:06.926193 29995 leader_election.cc:288] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:06.810367 29765 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:06.925850 30216 consensus_queue.cc:237] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:06.926936 29608 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:06.929864 26984 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50942 (request call id 465) took 5438 ms. Trace:
I20260807 11:11:06.874753 30124 raft_consensus.cc:729] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:06.922024 25750 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:06.876705 30320 consensus_queue.cc:1035] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.059s
I20260807 11:11:06.948114 30126 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:06.816583 30298 raft_consensus.cc:504] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:06.869083 30393 consensus_queue.cc:1035] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.145s
W20260807 11:11:06.962963 30401 log.cc:924] Time spent T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.175s user 0.001s sys 0.000s
I20260807 11:11:06.962766 30378 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:06.935886 30030 raft_consensus.cc:2726] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 11:11:06.973385 26035 raft_consensus.cc:1261] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 11:11:06.970783 26984 rpcz_store.cc:276] 0807 11:11:01.491008 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:01.493851 (+ 2843us) service_pool.cc:227] Handling call
0807 11:11:01.705293 (+211442us) spinlock_profiling.cc:243] Waited 1.41 ms on lock 0x7b5800000f98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc461a2b0 00007f3fc46057f9 00007f3fc46052a2 00007f3fbdceb684
0807 11:11:01.705542 (+ 249us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:01.705831 (+ 289us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:01.705865 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:01.706453 (+ 588us) log.cc:842] Serialized 26 byte log entry
0807 11:11:01.827306 (+120853us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:01.827352 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:01.827430 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:06.740002 (+4912572us) spinlock_profiling.cc:243] Waited 3.12 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:11:06.929589 (+189587us) raft_consensus.cc:1671] finished
0807 11:11:06.929620 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:06.929787 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10425472,"thread_start_us":120341,"threads_started":1,"wal-append.queue_time_us":168559}
I20260807 11:11:06.974887 25801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1837) took 2220 ms. Trace:
I20260807 11:11:06.975070 26036 raft_consensus.cc:1226] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:06.896608 29966 raft_consensus.cc:504] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:06.979555 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:06.949571 29323 raft_consensus.cc:3037] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:06.968580 29654 raft_consensus.cc:2781] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:11:06.877641 30383 consensus_queue.cc:1035] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.100s
I20260807 11:11:06.992982 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:07.006337 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:07.007385 25916 pending_rounds.cc:76] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc: Aborting all ops after (but not including) 7
I20260807 11:11:07.004200 26035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1905) took 2659 ms. Trace:
I20260807 11:11:07.026453 25916 pending_rounds.cc:97] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260807 11:11:07.004974 25801 rpcz_store.cc:276] 0807 11:11:04.754514 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.725428 (+1970914us) service_pool.cc:227] Handling call
0807 11:11:06.725731 (+ 303us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:06.725926 (+ 195us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 11:11:06.881497 (+155571us) log.cc:842] Serialized 14 byte log entry
0807 11:11:06.974532 (+ 93035us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:06.974599 (+ 67us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:06.974658 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:06.974725 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:06.974833 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15783,"raft.queue_time_us":5581,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":7535,"thread_start_us":242465,"threads_started":2}
I20260807 11:11:07.026778 25916 raft_consensus.cc:2866] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:11:07.023555 26042 raft_consensus.cc:1261] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260807 11:11:07.027706 26042 pending_rounds.cc:76] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9: Aborting all ops after (but not including) 3
I20260807 11:11:07.028162 26042 pending_rounds.cc:97] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260807 11:11:07.016696 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:07.028656 25801 raft_consensus.cc:1226] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260807 11:11:07.028959 25925 raft_consensus.cc:2454] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:11:07.026600 26035 rpcz_store.cc:276] 0807 11:11:04.345033 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.737072 (+2392039us) service_pool.cc:227] Handling call
0807 11:11:06.737437 (+ 365us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:07.003836 (+266399us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:07.004110 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:07.028600 26042 raft_consensus.cc:2866] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260807 11:11:06.987293 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 65570.3us
I20260807 11:11:07.032346 25750 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:11:07.030471 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:11:06.998268 26936 connection.cc:232] RPC call timeout handler was delayed by -1.1255098399706185s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1531s for a response
I20260807 11:11:07.033319 26035 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:07.034991 25750 leader_election.cc:302] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9
I20260807 11:11:06.949635 29680 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:07.037834 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:07.041647 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:07.045713 25808 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Advancing to term 9
I20260807 11:11:07.044893 26036 raft_consensus.cc:2454] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:11:07.047561 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1818) took 1612 ms. Trace:
I20260807 11:11:07.048067 25752 leader_election.cc:302] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:07.063263 25750 leader_election.cc:302] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:07.063427 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:07.064749 26046 raft_consensus.cc:2396] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260807 11:11:07.068044 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:07.068993 25920 raft_consensus.cc:2396] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260807 11:11:07.002770 30108 raft_consensus.cc:2726] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 11:11:07.071990 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:07.021708 29654 raft_consensus.cc:482] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:07.018195 29950 raft_consensus.cc:2741] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260807 11:11:07.073798 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:06.933158 30397 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:07.100360 26046 raft_consensus.cc:3037] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:07.001957 30296 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:07.024439 30153 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:07.112214 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:07.061810 29939 consensus_queue.cc:1035] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.076s
I20260807 11:11:07.126467 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: true dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:07.139585 25918 raft_consensus.cc:3032] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 5 LEADER]: Stepping down as leader of term 5
I20260807 11:11:07.139997 25918 raft_consensus.cc:729] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:07.126319 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:07.127576 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:11:07.128060 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 64869us
I20260807 11:11:07.061996 30283 raft_consensus.cc:2781] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:11:06.997939 30404 consensus_queue.cc:1035] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.087s
I20260807 11:11:07.132658 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:07.133150 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:07.132678 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:06.994645 29608 consensus_queue.cc:1035] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.090s
I20260807 11:11:07.141857 25750 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:07.178812 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:07.083210 29980 raft_consensus.cc:2781] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:11:07.180338 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:07.015152 30130 consensus_queue.cc:1035] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.095s
I20260807 11:11:07.149878 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:07.048195 26042 rpcz_store.cc:276] 0807 11:11:05.435122 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.860091 (+1424969us) service_pool.cc:227] Handling call
0807 11:11:06.881743 (+ 21652us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:07.047136 (+165393us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:07.047476 (+ 340us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:07.195087 25920 raft_consensus.cc:3037] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:07.195611 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:07.196491 26042 raft_consensus.cc:2454] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 9.
I20260807 11:11:07.153769 25918 consensus_queue.cc:260] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:07.197826 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:07.198315 25918 raft_consensus.cc:3037] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:07.198622 26042 raft_consensus.cc:2454] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:11:07.153270 26035 raft_consensus.cc:2454] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 8.
I20260807 11:11:07.154148 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:07.200974 26035 raft_consensus.cc:3032] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:11:07.201422 26035 raft_consensus.cc:729] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:11:07.202373 26035 consensus_queue.cc:260] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:07.203647 26035 raft_consensus.cc:3037] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:11:07.207227 25752 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:07.208461 25752 leader_election.cc:302] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:07.209807 25752 leader_election.cc:302] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:07.154214 25808 raft_consensus.cc:2454] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 9.
W20260807 11:11:07.163225 30417 log.cc:924] Time spent T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260807 11:11:07.172712 29966 leader_election.cc:288] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:07.065073 30124 consensus_queue.cc:260] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:07.223615 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:07.226213 30276 raft_consensus.cc:2781] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 11:11:07.226809 30276 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:07.227206 30276 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Advancing to term 10
I20260807 11:11:07.169193 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:07.229207 25797 raft_consensus.cc:2454] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 5.
I20260807 11:11:07.170428 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:07.145473 30378 raft_consensus.cc:3037] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
W20260807 11:11:07.231352 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 246819us
I20260807 11:11:07.173756 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:07.180546 29765 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:06.996592 30407 consensus_queue.cc:1046] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.163s
I20260807 11:11:07.232597 30184 consensus_queue.cc:1035] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:11:07.233716 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:07.234668 25925 raft_consensus.cc:2379] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 5: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:11:07.234798 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 345) took 2369 ms. Trace:
I20260807 11:11:07.235529 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:07.181095 29980 raft_consensus.cc:686] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:07.236629 29608 consensus_queue.cc:1035] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:11:07.236827 30184 consensus_queue.cc:1035] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 11:11:07.182117 29995 raft_consensus.cc:2726] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:07.237389 30424 raft_consensus.cc:3032] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:11:07.237951 30424 raft_consensus.cc:729] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:07.238582 25872 leader_election.cc:302] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:07.238931 30424 consensus_queue.cc:260] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:07.223606 29680 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:07.231388 26036 raft_consensus.cc:3032] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 11:11:07.242403 26036 raft_consensus.cc:729] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
W20260807 11:11:07.182228 30411 log.cc:924] Time spent T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260807 11:11:07.182530 30085 raft_consensus.cc:482] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:07.183811 25786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 396) took 3909 ms. Trace:
I20260807 11:11:07.243027 25786 rpcz_store.cc:276] 0807 11:11:03.274269 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.460166 (+3185897us) service_pool.cc:227] Handling call
0807 11:11:06.460492 (+ 326us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:06.698532 (+238040us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:07.183477 (+484945us) spinlock_profiling.cc:243] Waited 237 ms on lock 0x7b54003acad4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:07.183729 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":545023360}
I20260807 11:11:07.146323 26037 raft_consensus.cc:1261] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260807 11:11:07.185964 25798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1827) took 4636 ms. Trace:
I20260807 11:11:07.043038 30416 consensus_queue.cc:1046] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.158s
I20260807 11:11:07.145449 30391 leader_election.cc:288] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:07.174782 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:11:07.194000 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 52248.9us
I20260807 11:11:07.125607 30309 consensus_queue.cc:1035] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.091s
I20260807 11:11:07.218122 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:07.107969 29654 raft_consensus.cc:3037] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:07.166496 30344 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:07.240682 30424 raft_consensus.cc:3037] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:11:07.258508 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:07.259449 25919 raft_consensus.cc:2454] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 5.
I20260807 11:11:07.245095 25798 rpcz_store.cc:276] 0807 11:11:02.549596 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.433908 (+3884312us) service_pool.cc:227] Handling call
0807 11:11:06.434210 (+ 302us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:06.702666 (+268456us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:07.185696 (+483030us) spinlock_profiling.cc:243] Waited 267 ms on lock 0x7b54000e4cd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:07.185907 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":615142784}
I20260807 11:11:07.245617 25872 leader_election.cc:302] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:07.145401 30283 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:07.248176 25925 raft_consensus.cc:1226] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6. Ops: []
I20260807 11:11:07.252738 26036 consensus_queue.cc:260] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:07.268317 26036 raft_consensus.cc:3037] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:07.268584 25918 raft_consensus.cc:2454] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 6.
I20260807 11:11:07.235304 25917 rpcz_store.cc:276] 0807 11:11:04.865246 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.955962 (+ 90716us) service_pool.cc:227] Handling call
0807 11:11:04.960480 (+ 4518us) raft_consensus.cc:1457] Updating replica for 3 ops
0807 11:11:04.986805 (+ 26325us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:04.986836 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0807 11:11:05.716880 (+730044us) log.cc:842] Serialized 624 byte log entry
0807 11:11:06.672263 (+955383us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:07.196488 (+524225us) log.cc:842] Serialized 14 byte log entry
0807 11:11:07.234382 (+ 37894us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:07.234480 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:07.234542 (+ 62us) raft_consensus.cc:1671] finished
0807 11:11:07.234559 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:07.234727 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34953,"raft.queue_time_us":60159,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":7730,"thread_start_us":820214,"threads_started":1,"wal-append.queue_time_us":921859}
I20260807 11:11:07.253015 25750 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:07.269713 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:07.270895 25750 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:11:07.253454 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 43748.9us
W20260807 11:11:07.256918 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 190216us
I20260807 11:11:07.261034 25872 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:11:07.246428 28964 raft_consensus.cc:2781] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:07.275624 25872 leader_election.cc:302] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:07.281157 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:07.230962 26046 raft_consensus.cc:2454] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 7.
I20260807 11:11:07.279928 26035 raft_consensus.cc:1261] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
W20260807 11:11:07.254796 30105 consensus_meta.cc:287] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.692s user 0.012s sys 0.000s
I20260807 11:11:07.282400 26035 pending_rounds.cc:76] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9: Aborting all ops after (but not including) 5
I20260807 11:11:07.270644 25919 raft_consensus.cc:2454] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 7.
I20260807 11:11:07.287099 26035 pending_rounds.cc:97] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260807 11:11:07.287741 26035 raft_consensus.cc:2866] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:11:07.283649 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:07.276326 25920 raft_consensus.cc:2454] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 7.
I20260807 11:11:07.284718 25752 leader_election.cc:302] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:07.245291 30184 raft_consensus.cc:2726] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:11:07.301846 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:07.313613 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:07.315409 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:07.316223 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:07.316493 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:07.316831 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:07.254624 30085 raft_consensus.cc:504] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:07.314670 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:07.319651 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:07.320663 25918 raft_consensus.cc:2454] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 6.
W20260807 11:11:07.261874 30366 log.cc:924] Time spent T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 1.131s user 0.004s sys 0.004s
I20260807 11:11:07.311256 25872 leader_election.cc:302] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:07.306594 25750 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:07.330266 25872 leader_election.cc:302] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:11:07.302466 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1912) took 2478 ms. Trace:
W20260807 11:11:07.331797 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 47223.8us
I20260807 11:11:07.317044 25920 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 9.
I20260807 11:11:07.257182 29680 raft_consensus.cc:2781] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 11:11:07.317082 25798 raft_consensus.cc:3037] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:11:07.288867 26936 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
W20260807 11:11:07.340579 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 51818.7us
I20260807 11:11:07.341511 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: true dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:07.342288 25797 raft_consensus.cc:3037] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:07.275605 30344 raft_consensus.cc:2726] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:11:07.344512 25872 leader_election.cc:302] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:07.346467 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:11:07.347523 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 411708us
I20260807 11:11:07.334546 25932 raft_consensus.cc:1261] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260807 11:11:07.318923 25917 raft_consensus.cc:2454] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 6.
I20260807 11:11:07.331198 26039 rpcz_store.cc:276] 0807 11:11:04.824360 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.793977 (+1969617us) service_pool.cc:227] Handling call
0807 11:11:06.794401 (+ 424us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:06.873037 (+ 78636us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:06.873081 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:06.920029 (+ 46948us) log.cc:842] Serialized 26 byte log entry
0807 11:11:07.010882 (+ 90853us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:11:07.010934 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:07.011036 (+ 102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:07.302262 (+291226us) raft_consensus.cc:1671] finished
0807 11:11:07.302288 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:07.302413 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16190,"raft.queue_time_us":65954,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":10079,"thread_start_us":77681,"threads_started":1,"wal-append.queue_time_us":367530}
I20260807 11:11:07.371778 26936 leader_election.cc:302] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:07.337281 25931 raft_consensus.cc:2396] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260807 11:11:07.275279 30283 raft_consensus.cc:3037] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:07.349097 30407 raft_consensus.cc:2781] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:07.349436 25920 raft_consensus.cc:1226] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5. Ops: [3.2-4.3]
W20260807 11:11:07.354884 30440 log.cc:924] Time spent T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20260807 11:11:07.311939 30441 raft_consensus.cc:2781] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 11:11:07.355629 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:07.356431 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:07.336544 29608 raft_consensus.cc:2726] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:07.291149 30258 raft_consensus.cc:2726] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:11:07.285804 29995 raft_consensus.cc:2726] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 11:11:07.237478 29980 consensus_queue.cc:237] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:07.358109 30344 raft_consensus.cc:2726] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:07.358556 29680 raft_consensus.cc:686] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 8 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:07.360793 30085 leader_election.cc:288] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:07.314121 30393 consensus_queue.cc:1035] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.069s
I20260807 11:11:07.341683 30415 raft_consensus.cc:2781] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader election won for term 7
W20260807 11:11:07.294656 30048 consensus_meta.cc:287] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.645s user 0.009s sys 0.004s
I20260807 11:11:07.333923 25750 leader_election.cc:302] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:11:07.286131 28964 raft_consensus.cc:482] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 11:11:07.303763 30152 consensus_meta.cc:287] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.538s user 0.009s sys 0.003s
I20260807 11:11:07.366688 25798 raft_consensus.cc:2454] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:11:07.428715 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:07.431003 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:07.432442 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:07.363860 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:11:07.367409 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 283777us
I20260807 11:11:07.303078 29972 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:07.360066 30424 consensus_queue.cc:1035] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20260807 11:11:07.372577 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:07.381299 25919 raft_consensus.cc:1226] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6. Ops: [3.2-5.4]
I20260807 11:11:07.381843 25797 raft_consensus.cc:2454] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 6.
I20260807 11:11:07.304145 30439 raft_consensus.cc:2781] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader election won for term 7
I20260807 11:11:07.386583 26936 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:07.343533 30383 raft_consensus.cc:2726] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 11:11:07.339313 30298 leader_election.cc:288] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:07.394505 25918 raft_consensus.cc:1226] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6. Ops: [3.3-5.4]
I20260807 11:11:07.363022 30154 raft_consensus.cc:2781] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:11:07.301604 30391 consensus_queue.cc:1046] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.073s
I20260807 11:11:07.370736 30397 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:07.419767 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:11:07.426189 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 92331.4us
I20260807 11:11:07.433710 30102 leader_election.cc:288] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:07.392477 30184 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:07.382331 30407 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:07.440330 26979 raft_consensus.cc:1261] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260807 11:11:07.400840 30171 raft_consensus.cc:2726] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 11:11:07.444005 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:07.447326 30130 raft_consensus.cc:2726] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260807 11:11:07.458974 30444 log.cc:924] Time spent T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260807 11:11:07.442708 26039 raft_consensus.cc:2454] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
W20260807 11:11:07.484906 25981 connection.cc:232] RPC call timeout handler was delayed by -9.5976521580014378s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1827s for a response
I20260807 11:11:07.373358 30182 consensus_queue.cc:1046] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.060s
I20260807 11:11:07.391521 30441 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:07.434213 28964 raft_consensus.cc:3037] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:07.465628 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:07.468227 25750 leader_election.cc:302] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:07.427225 25869 leader_election.cc:302] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:07.470594 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:11:07.456662 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 70203.3us
I20260807 11:11:07.474792 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:07.426206 30344 consensus_queue.cc:1046] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260807 11:11:07.445604 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:07.373659 30404 consensus_queue.cc:1035] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260807 11:11:07.434671 30048 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:07.453070 30421 consensus_queue.cc:1035] T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.081s
I20260807 11:11:07.455323 29680 consensus_queue.cc:237] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:07.479633 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 506) took 2612 ms. Trace:
I20260807 11:11:07.558275 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:07.591878 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 344) took 2729 ms. Trace:
I20260807 11:11:07.604816 25932 rpcz_store.cc:276] 0807 11:11:04.862390 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.930899 (+ 68509us) service_pool.cc:227] Handling call
0807 11:11:05.032705 (+101806us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:07.371703 (+2338998us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:07.591581 (+219878us) spinlock_profiling.cc:243] Waited 2.3 s on lock 0x7b54003633d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:07.591805 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5292681088}
I20260807 11:11:07.514603 25799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1839) took 2658 ms. Trace:
I20260807 11:11:07.555747 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 1918) took 1062 ms. Trace:
I20260807 11:11:07.556030 26041 raft_consensus.cc:1261] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 11:11:07.604581 25798 raft_consensus.cc:2379] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 7: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:11:07.605432 26035 raft_consensus.cc:2454] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:11:07.603720 26033 rpcz_store.cc:276] 0807 11:11:04.867270 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.795046 (+1927776us) service_pool.cc:227] Handling call
0807 11:11:06.795329 (+ 283us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:06.795602 (+ 273us) raft_consensus.cc:1520] Early marking committed up to index 7
0807 11:11:06.868459 (+ 72857us) log.cc:842] Serialized 14 byte log entry
0807 11:11:07.367845 (+499386us) log.cc:842] Serialized 14 byte log entry
0807 11:11:07.479287 (+111442us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:07.479344 (+ 57us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:07.479393 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:07.479455 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:07.479558 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14793,"prepare.queue_time_us":275455,"prepare.run_cpu_time_us":109,"prepare.run_wall_time_us":4077,"raft.queue_time_us":100950,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":13097,"spinlock_wait_cycles":10606848,"thread_start_us":592977,"threads_started":3,"wal-append.queue_time_us":239753}
I20260807 11:11:07.514211 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:07.612536 26034 raft_consensus.cc:2379] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 7: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:11:07.619473 25872 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:07.619884 26039 raft_consensus.cc:2454] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:11:07.620071 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:11:07.611922 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 184807us
I20260807 11:11:07.606896 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:07.607661 26038 rpcz_store.cc:276] 0807 11:11:06.493599 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.700553 (+206954us) service_pool.cc:227] Handling call
0807 11:11:07.555560 (+855007us) spinlock_profiling.cc:243] Waited 9.88 ms on lock 0x7b540009f454. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:07.555691 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22713088}
I20260807 11:11:07.621369 25869 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:07.621946 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:07.619513 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:07.623766 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:07.609632 25803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1840) took 2753 ms. Trace:
I20260807 11:11:07.625097 26038 raft_consensus.cc:3032] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 11:11:07.625720 26038 raft_consensus.cc:729] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:11:07.629642 26039 raft_consensus.cc:1226] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:07.624714 25803 rpcz_store.cc:276] 0807 11:11:04.856466 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.733510 (+1877044us) service_pool.cc:227] Handling call
0807 11:11:06.733782 (+ 272us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:06.920983 (+187201us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 11:11:07.023480 (+102497us) log.cc:842] Serialized 14 byte log entry
0807 11:11:07.274257 (+250777us) log.cc:842] Serialized 14 byte log entry
0807 11:11:07.362350 (+ 88093us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:07.362414 (+ 64us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:07.362459 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:07.597971 (+235512us) spinlock_profiling.cc:243] Waited 4.38 ms on lock 0x7b18002e9600. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbc769f96 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9
0807 11:11:07.609289 (+ 11318us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b54000483d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:07.609361 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:07.609521 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":130976,"prepare.queue_time_us":225019,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":94,"raft.queue_time_us":88079,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":7530,"spinlock_wait_cycles":445513216,"thread_start_us":322593,"threads_started":2,"wal-append.queue_time_us":387223}
I20260807 11:11:07.474360 30309 raft_consensus.cc:2781] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 FOLLOWER]: Leader election won for term 9
I20260807 11:11:07.635268 25869 leader_election.cc:302] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:07.633383 26984 raft_consensus.cc:1226] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 6. Current term is 7. Ops: [6.8-6.8]
W20260807 11:11:07.724464 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 89303us
I20260807 11:11:07.724505 30452 consensus_queue.cc:1035] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.273s
I20260807 11:11:07.586412 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:07.726145 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1819) took 2290 ms. Trace:
I20260807 11:11:07.612104 30441 raft_consensus.cc:3037] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:07.628465 25808 raft_consensus.cc:2454] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 7.
W20260807 11:11:07.725167 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 100484us
I20260807 11:11:07.726558 26037 rpcz_store.cc:276] 0807 11:11:05.435545 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.864101 (+1428556us) service_pool.cc:227] Handling call
0807 11:11:06.893440 (+ 29339us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:07.244276 (+350836us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:07.725870 (+481594us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b54000bb154. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:07.726077 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":580495744}
I20260807 11:11:07.703693 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:11:07.554533 30464 log.cc:924] Time spent T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20260807 11:11:07.607059 25799 rpcz_store.cc:276] 0807 11:11:04.855881 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.727715 (+1871834us) service_pool.cc:227] Handling call
0807 11:11:06.728049 (+ 334us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:06.740206 (+ 12157us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:06.740234 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:06.741616 (+ 1382us) log.cc:842] Serialized 26 byte log entry
0807 11:11:06.924488 (+182872us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:11:07.389861 (+465373us) log.cc:842] Serialized 14 byte log entry
0807 11:11:07.513458 (+123597us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:07.513778 (+ 320us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:07.513941 (+ 163us) raft_consensus.cc:1671] finished
0807 11:11:07.513953 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:07.514459 (+ 506us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":48605,"prepare.queue_time_us":252378,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":35925,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":6424,"thread_start_us":588110,"threads_started":3,"wal-append.queue_time_us":404275}
I20260807 11:11:07.634444 25872 leader_election.cc:302] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:07.745572 26039 raft_consensus.cc:3037] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:07.735390 26038 consensus_queue.cc:260] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:07.677189 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:07.750809 26038 raft_consensus.cc:3037] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:07.730868 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:07.684351 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:07.483210 30439 raft_consensus.cc:686] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [term 7 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:07.661971 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:07.483922 30407 raft_consensus.cc:3037] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:07.512251 30397 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:11:07.557139 30162 consensus_meta.cc:287] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.584s user 0.010s sys 0.001s
W20260807 11:11:07.593927 30316 log.cc:924] Time spent T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 3.062s user 0.000s sys 0.029s
I20260807 11:11:07.749475 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:07.720450 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:11:07.515799 29692 consensus_meta.cc:287] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.873s user 0.013s sys 0.001s
I20260807 11:11:07.645562 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:11:07.757161 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 122774us
I20260807 11:11:07.644073 30378 raft_consensus.cc:2726] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:11:07.733999 30404 raft_consensus.cc:2726] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:07.761658 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:07.584234 30112 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:07.633735 30247 raft_consensus.cc:2781] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260807 11:11:07.683751 30172 consensus_meta.cc:287] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.742s user 0.012s sys 0.002s
I20260807 11:11:07.753350 30309 raft_consensus.cc:686] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:07.632741 30048 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:07.763283 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:07.765264 25981 leader_election.cc:302] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:07.765897 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:11:07.819373 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 60373.3us
I20260807 11:11:07.716485 30030 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:07.633437 30344 raft_consensus.cc:3032] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Stepping down as leader of term 6
I20260807 11:11:07.623796 30270 raft_consensus.cc:2726] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 11:11:07.513334 30184 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:07.789454 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:07.862097 25799 raft_consensus.cc:2396] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 11:11:07.763427 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:07.634635 29847 raft_consensus.cc:2726] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 11:11:07.806958 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:07.764628 30126 raft_consensus.cc:2781] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:11:07.703658 30182 consensus_queue.cc:1035] T 66d10b6b12404f0488b9f83c8080069f P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.085s
W20260807 11:11:07.824434 26036 consensus_meta.cc:287] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.555s user 0.008s sys 0.002s
I20260807 11:11:07.824682 25981 leader_election.cc:302] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:07.825225 26038 raft_consensus.cc:1261] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 11:11:07.877302 25931 raft_consensus.cc:1226] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: []
I20260807 11:11:07.826690 30462 consensus_queue.cc:1035] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.021s
W20260807 11:11:07.827370 30276 consensus_meta.cc:287] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.600s user 0.012s sys 0.000s
I20260807 11:11:07.793694 30247 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:07.790208 30397 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:07.823405 30112 raft_consensus.cc:2741] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20260807 11:11:07.830462 29323 consensus_meta.cc:287] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.807s user 0.005s sys 0.010s
I20260807 11:11:07.823962 30404 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:07.831529 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:07.851171 26035 raft_consensus.cc:3037] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 7
I20260807 11:11:07.851594 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:11:07.821168 29654 consensus_meta.cc:287] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.549s user 0.013s sys 0.006s
I20260807 11:11:07.635468 30338 consensus_queue.cc:1035] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.170s
W20260807 11:11:07.827406 30464 log.cc:924] Time spent T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.061s user 0.000s sys 0.001s
I20260807 11:11:07.831162 30309 consensus_queue.cc:237] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:07.867733 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:07.873750 26979 raft_consensus.cc:2354] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 7. Current term is 8.
I20260807 11:11:07.833216 30344 raft_consensus.cc:729] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:07.876456 26037 raft_consensus.cc:1226] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:07.788702 30162 raft_consensus.cc:504] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:07.884001 26036 raft_consensus.cc:1261] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
W20260807 11:11:07.877970 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 53374.2us
I20260807 11:11:07.794885 29692 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:07.887612 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:07.833038 30172 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:07.856153 30030 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:08.264449 26036 pending_rounds.cc:76] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9: Aborting all ops after (but not including) 3
I20260807 11:11:08.267354 26036 pending_rounds.cc:97] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260807 11:11:08.267812 26036 raft_consensus.cc:2866] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:11:08.268719 26036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1921) took 1269 ms. Trace:
I20260807 11:11:08.269213 26036 rpcz_store.cc:276] 0807 11:11:06.999675 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:07.230883 (+231208us) service_pool.cc:227] Handling call
0807 11:11:07.231276 (+ 393us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:08.268424 (+1037148us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:08.268659 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:08.265746 26986 raft_consensus.cc:2396] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
W20260807 11:11:08.266475 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 408075us
I20260807 11:11:07.889617 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:08.094143 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:08.282938 26035 raft_consensus.cc:1261] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260807 11:11:08.281754 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:08.256387 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:08.285424 25869 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
W20260807 11:11:08.287916 26039 consensus_meta.cc:287] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.538s user 0.010s sys 0.002s
I20260807 11:11:08.288820 26987 raft_consensus.cc:2354] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6.
I20260807 11:11:08.288486 26039 raft_consensus.cc:2454] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 7.
I20260807 11:11:08.291395 26938 leader_election.cc:302] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:11:07.941437 29883 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
W20260807 11:11:08.241834 28964 consensus_meta.cc:287] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.606s user 0.013s sys 0.000s
W20260807 11:11:07.951035 30283 consensus_meta.cc:287] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.554s user 0.011s sys 0.000s
I20260807 11:11:08.301364 25981 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:07.884577 30351 raft_consensus.cc:2726] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:11:07.878440 30126 raft_consensus.cc:482] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:08.301844 25869 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
W20260807 11:11:07.937104 30481 log.cc:924] Time spent T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20260807 11:11:08.313345 26985 raft_consensus.cc:2354] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 6. Current term is 7.
I20260807 11:11:07.945191 29966 raft_consensus.cc:3032] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Stepping down as leader of term 6
W20260807 11:11:07.954435 30484 log.cc:924] Time spent T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20260807 11:11:08.329023 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:07.913733 30247 raft_consensus.cc:3037] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:08.316522 26979 raft_consensus.cc:1226] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 LEADER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5. Ops: []
I20260807 11:11:08.086316 26974 raft_consensus.cc:1226] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 3. Current term is 4. Ops: []
I20260807 11:11:07.940780 30154 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:08.322158 25981 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:08.329514 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:08.330535 26978 raft_consensus.cc:2379] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 7: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:11:08.331410 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:08.335040 26979 raft_consensus.cc:2354] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 6. Current term is 7.
I20260807 11:11:08.313800 30232 raft_consensus.cc:2781] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 11:11:07.885403 29847 consensus_queue.cc:1046] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.437s
I20260807 11:11:08.302703 30449 raft_consensus.cc:2726] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:08.345252 29883 consensus_queue.cc:1035] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:11:08.346096 29883 consensus_queue.cc:1225] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer b22d69db45194908ad3d345eba9badd9 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260807 11:11:08.346463 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:08.347369 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:08.347786 30449 consensus_queue.cc:1046] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 11:11:08.299240 30105 consensus_queue.cc:1035] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.047s
I20260807 11:11:07.885165 30439 consensus_queue.cc:237] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:08.349349 30496 raft_consensus.cc:3032] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Stepping down as leader of term 6
I20260807 11:11:08.351008 30496 raft_consensus.cc:729] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:08.322902 30283 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:08.291642 30404 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:08.351847 30496 consensus_queue.cc:260] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:08.330757 30154 raft_consensus.cc:3037] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:08.349324 26974 raft_consensus.cc:2454] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 7.
I20260807 11:11:08.354049 26976 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 481) took 5804 ms. Trace:
I20260807 11:11:08.354239 30030 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:07.914286 30276 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:08.328291 30126 raft_consensus.cc:3037] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:08.328015 29966 raft_consensus.cc:729] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:08.314816 28964 raft_consensus.cc:504] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:07.918982 29323 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:08.292387 29654 raft_consensus.cc:504] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:08.332527 30184 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:11:08.413547 30407 consensus_meta.cc:287] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.623s user 0.009s sys 0.001s
I20260807 11:11:08.413954 30397 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:08.414059 30407 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:08.415613 29654 leader_election.cc:288] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:08.370736 30232 raft_consensus.cc:686] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:08.405818 30171 raft_consensus.cc:482] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:08.413547 30283 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:11:08.437803 30379 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:11:04.923090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.120.132:43279 (local address 127.0.0.1:41764)
0807 11:11:06.608890 (+1685800us) negotiation.cc:107] Waiting for socket to connect
0807 11:11:06.608989 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0807 11:11:06.609682 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:11:07.344933 (+735251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:11:07.356941 (+ 12008us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:11:07.357126 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:11:07.357963 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:11:07.358061 (+ 98us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:11:07.568397 (+210336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:11:07.568459 (+ 62us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:11:07.634516 (+ 66057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:11:07.634581 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:11:07.759470 (+124889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:11:07.762371 (+ 2901us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:11:07.829255 (+ 66884us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:11:07.853499 (+ 24244us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:11:07.853537 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:11:07.853617 (+ 80us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:11:07.853839 (+ 222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:11:07.861392 (+ 7553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:11:07.865256 (+ 3864us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:11:07.876703 (+ 11447us) client_negotiation.cc:769] Sending connection context
0807 11:11:08.040418 (+163715us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.23.120.132:43279: sent 0 of 16 requested bytes
Metrics: {"client-negotiator.queue_time_us":1637680,"thread_start_us":1684130,"threads_started":1}
I20260807 11:11:08.433794 30407 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:11:08.443811 30426 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:11:06.765162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41764 (local address 127.23.120.132:43279)
0807 11:11:07.309341 (+544179us) server_negotiation.cc:206] Beginning negotiation
0807 11:11:07.309387 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0807 11:11:07.309464 (+ 77us) server_negotiation.cc:407] Connection header received
0807 11:11:07.309632 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:11:07.309678 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:11:07.310003 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:11:07.310250 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:11:07.360638 (+ 50388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:11:07.568003 (+207365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:11:07.634933 (+ 66930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:11:07.636942 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:11:07.637481 (+ 539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:11:07.854437 (+216956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:11:07.854528 (+ 91us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:11:07.854577 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:11:07.854714 (+ 137us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:11:07.854849 (+ 135us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:11:07.854923 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:11:07.855115 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0807 11:11:08.071977 (+216862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41764: BlockingRecv error: recv got EOF from 127.0.0.1:41764 (error 108)
Metrics: {"server-negotiator.queue_time_us":496609,"thread_start_us":184977,"threads_started":1}
W20260807 11:11:08.443789 30441 consensus_meta.cc:287] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.717s user 0.015s sys 0.000s
I20260807 11:11:08.443717 30404 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:08.445137 30441 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:08.477046 25797 raft_consensus.cc:2354] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6.
I20260807 11:11:08.315608 25799 raft_consensus.cc:1226] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6. Ops: []
I20260807 11:11:08.479743 25799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53324 (request call id 1591) took 1171 ms. Trace:
I20260807 11:11:08.480341 25799 rpcz_store.cc:276] 0807 11:11:07.307825 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.288711 (+980886us) service_pool.cc:227] Handling call
0807 11:11:08.315416 (+ 26705us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:08.478813 (+163397us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:08.479487 (+ 674us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:08.481432 25798 raft_consensus.cc:1226] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [term 3 LEADER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 2. Current term is 3. Ops: []
I20260807 11:11:08.483131 25798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53324 (request call id 1594) took 1174 ms. Trace:
I20260807 11:11:08.483400 25799 raft_consensus.cc:1226] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260807 11:11:08.483676 25798 rpcz_store.cc:276] 0807 11:11:07.308732 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.479293 (+1170561us) service_pool.cc:227] Handling call
0807 11:11:08.480671 (+ 1378us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:08.482713 (+ 2042us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:08.483030 (+ 317us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:08.484371 25799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53324 (request call id 1595) took 1135 ms. Trace:
I20260807 11:11:08.484876 25799 rpcz_store.cc:276] 0807 11:11:07.349230 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.481953 (+1132723us) service_pool.cc:227] Handling call
0807 11:11:08.482732 (+ 779us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:08.484028 (+ 1296us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:08.484290 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:08.292877 25801 raft_consensus.cc:1226] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6. Ops: []
I20260807 11:11:08.486878 25801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53324 (request call id 1593) took 1178 ms. Trace:
I20260807 11:11:08.487326 25801 rpcz_store.cc:276] 0807 11:11:07.308436 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.291979 (+983543us) service_pool.cc:227] Handling call
0807 11:11:08.292537 (+ 558us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:08.486252 (+193715us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:08.486661 (+ 409us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:08.490603 25803 raft_consensus.cc:2354] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 6. Current term is 7.
W20260807 11:11:08.493037 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 191121us
I20260807 11:11:08.497298 25869 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
W20260807 11:11:08.502710 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 180421us
I20260807 11:11:08.502863 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1934) took 1169 ms. Trace:
I20260807 11:11:08.503593 26034 rpcz_store.cc:276] 0807 11:11:07.333389 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:07.620824 (+287435us) service_pool.cc:227] Handling call
0807 11:11:07.704536 (+ 83712us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:07.734157 (+ 29621us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:07.734209 (+ 52us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:07.753274 (+ 19065us) log.cc:842] Serialized 26 byte log entry
0807 11:11:08.267015 (+513741us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:11:08.267056 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:08.267120 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:08.502079 (+234959us) raft_consensus.cc:1671] finished
0807 11:11:08.502123 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:08.502514 (+ 391us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31959,"raft.queue_time_us":42493,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":11885,"spinlock_wait_cycles":1788672,"thread_start_us":443531,"threads_started":1,"wal-append.queue_time_us":653823}
I20260807 11:11:08.506274 25916 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 350) took 1517 ms. Trace:
I20260807 11:11:08.508603 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 518) took 1073 ms. Trace:
I20260807 11:11:08.509032 26034 rpcz_store.cc:276] 0807 11:11:07.435407 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.507137 (+1071730us) service_pool.cc:227] Handling call
0807 11:11:08.507805 (+ 668us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:08.508261 (+ 456us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:08.508289 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:08.508382 (+ 93us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:08.508417 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:08.508485 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:08.508553 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:08.510831 26034 raft_consensus.cc:1226] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 7. Current term is 8. Ops: [5.4-7.5]
I20260807 11:11:08.511247 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1843) took 1068 ms. Trace:
I20260807 11:11:08.511435 26034 rpcz_store.cc:276] 0807 11:11:07.443263 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.509835 (+1066572us) service_pool.cc:227] Handling call
0807 11:11:08.510471 (+ 636us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:11:08.511093 (+ 622us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:08.511218 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:08.514871 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1836) took 1148 ms. Trace:
I20260807 11:11:08.506789 25916 rpcz_store.cc:276] 0807 11:11:06.988672 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.988835 (+ 163us) service_pool.cc:227] Handling call
0807 11:11:07.007168 (+ 18333us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:11:07.370563 (+363395us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:07.370594 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 11:11:07.371135 (+ 541us) log.cc:842] Serialized 52 byte log entry
0807 11:11:07.856118 (+484983us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:07.856153 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:07.856227 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:08.505625 (+649398us) raft_consensus.cc:1671] finished
0807 11:11:08.505665 (+ 40us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:08.505996 (+ 331us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11089,"raft.queue_time_us":123340,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":155,"thread_start_us":762239,"threads_started":2,"wal-append.queue_time_us":949326}
W20260807 11:11:08.517359 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 170723us
I20260807 11:11:08.354552 26976 rpcz_store.cc:276] 0807 11:11:02.549336 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:02.625738 (+ 76402us) service_pool.cc:227] Handling call
0807 11:11:02.853847 (+228109us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:03.278743 (+424896us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:07.769279 (+4490536us) spinlock_profiling.cc:243] Waited 5.05 ms on lock 0x7b5400512aa0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf654730 00007f3fbf6c7b42 00007f3fbf6c3f26 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772
0807 11:11:08.353783 (+584504us) spinlock_profiling.cc:243] Waited 85.2 ms on lock 0x7b540072f454. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:08.353983 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":207626880}
I20260807 11:11:08.521045 26986 raft_consensus.cc:2354] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6.
I20260807 11:11:08.515378 26046 rpcz_store.cc:276] 0807 11:11:07.366768 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:07.633762 (+266994us) service_pool.cc:227] Handling call
0807 11:11:07.634085 (+ 323us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:07.634525 (+ 440us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:07.634558 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:07.634997 (+ 439us) log.cc:842] Serialized 26 byte log entry
0807 11:11:07.866723 (+231726us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:07.866773 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:07.866854 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:08.514285 (+647431us) raft_consensus.cc:1671] finished
0807 11:11:08.514327 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:08.514642 (+ 315us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8271,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":139,"thread_start_us":231132,"threads_started":1,"wal-append.queue_time_us":692797}
I20260807 11:11:08.522899 29692 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195)
I20260807 11:11:08.522841 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:08.524230 26046 raft_consensus.cc:3037] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 6
I20260807 11:11:08.421329 30344 consensus_queue.cc:260] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:08.526592 25798 raft_consensus.cc:1226] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5. Ops: []
I20260807 11:11:08.525295 25799 raft_consensus.cc:1226] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5. Ops: []
W20260807 11:11:08.536015 30299 log.cc:924] Time spent T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 4.034s user 0.020s sys 0.043s
I20260807 11:11:08.536953 30162 leader_election.cc:288] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:08.538969 25806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1816) took 6577 ms. Trace:
I20260807 11:11:08.539574 30276 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 10 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:08.539394 25806 rpcz_store.cc:276] 0807 11:11:01.961737 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:03.183118 (+1221381us) service_pool.cc:227] Handling call
0807 11:11:03.183503 (+ 385us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:03.199623 (+ 16120us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:03.199660 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:03.215732 (+ 16072us) log.cc:842] Serialized 26 byte log entry
0807 11:11:03.713337 (+497605us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:03.713391 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:03.713484 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:08.419067 (+4705583us) spinlock_profiling.cc:243] Waited 3.98 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:11:08.538209 (+119142us) raft_consensus.cc:1671] finished
0807 11:11:08.538260 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:08.538711 (+ 451us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3523,"raft.queue_time_us":69974,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":9283,"spinlock_wait_cycles":9156864,"thread_start_us":468441,"threads_started":1,"wal-append.queue_time_us":1143909}
I20260807 11:11:08.545356 30232 consensus_queue.cc:237] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:08.548084 30441 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:08.534227 25798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53324 (request call id 1596) took 1183 ms. Trace:
I20260807 11:11:08.550077 25798 rpcz_store.cc:276] 0807 11:11:07.349633 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.513164 (+1163531us) service_pool.cc:227] Handling call
0807 11:11:08.513877 (+ 713us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:08.532828 (+ 18951us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:08.532965 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:08.534472 25799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53324 (request call id 1597) took 1183 ms. Trace:
W20260807 11:11:08.551944 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 242682us
I20260807 11:11:08.424467 30171 raft_consensus.cc:504] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:08.553789 30112 raft_consensus.cc:2726] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:11:08.554615 29323 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:08.555907 30171 leader_election.cc:288] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:08.553071 30172 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:08.555770 29966 consensus_queue.cc:260] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:08.554318 30500 consensus_queue.cc:1035] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260807 11:11:08.558552 29966 raft_consensus.cc:3037] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:08.565539 30184 raft_consensus.cc:2726] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:08.550791 25799 rpcz_store.cc:276] 0807 11:11:07.350018 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.513211 (+1163193us) service_pool.cc:227] Handling call
0807 11:11:08.513913 (+ 702us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:08.532740 (+ 18827us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:08.532955 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22784}
I20260807 11:11:08.572237 25797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1600) took 1221 ms. Trace:
I20260807 11:11:08.572798 25797 rpcz_store.cc:276] 0807 11:11:07.350983 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:07.645279 (+294296us) service_pool.cc:227] Handling call
0807 11:11:08.571880 (+926601us) spinlock_profiling.cc:243] Waited 703 ms on lock 0x7b54002f4c54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:08.572087 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1616081152}
I20260807 11:11:08.574747 30496 raft_consensus.cc:3037] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:08.577958 25803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1592) took 1269 ms. Trace:
I20260807 11:11:08.578434 25803 rpcz_store.cc:276] 0807 11:11:07.308128 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:07.683466 (+375338us) service_pool.cc:227] Handling call
0807 11:11:08.577704 (+894238us) spinlock_profiling.cc:243] Waited 724 ms on lock 0x7b540014e6d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:08.577872 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1665426176}
I20260807 11:11:08.581601 25797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1849) took 1159 ms. Trace:
I20260807 11:11:08.582072 25797 rpcz_store.cc:276] 0807 11:11:07.422261 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.575759 (+1153498us) service_pool.cc:227] Handling call
0807 11:11:08.576498 (+ 739us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:08.581062 (+ 4564us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:08.581104 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:08.581222 (+ 118us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:08.581277 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:08.581362 (+ 85us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:08.581511 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:08.584362 25797 raft_consensus.cc:1226] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4. Ops: []
I20260807 11:11:08.585309 25797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1853) took 1161 ms. Trace:
I20260807 11:11:08.585740 25797 rpcz_store.cc:276] 0807 11:11:07.423509 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.583352 (+1159843us) service_pool.cc:227] Handling call
0807 11:11:08.583964 (+ 612us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:08.584910 (+ 946us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:08.585152 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:08.591671 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:08.593142 25916 raft_consensus.cc:3037] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:08.603703 25916 raft_consensus.cc:2454] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:11:08.603744 30404 raft_consensus.cc:2726] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 11:11:08.605054 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:08.609711 30497 consensus_queue.cc:1035] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 11:11:08.610052 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1817) took 3177 ms. Trace:
I20260807 11:11:08.610592 26041 rpcz_store.cc:276] 0807 11:11:05.432934 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.815067 (+1382133us) service_pool.cc:227] Handling call
0807 11:11:07.555683 (+740616us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:07.620923 (+ 65240us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:08.609633 (+988710us) spinlock_profiling.cc:243] Waited 696 ms on lock 0x7b540009f450. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:08.609859 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1599722368}
I20260807 11:11:08.612025 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:08.614405 25981 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:08.614588 25797 raft_consensus.cc:1226] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 8. Current term is 9. Ops: [8.8-8.8]
W20260807 11:11:08.607266 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 47466.3us
I20260807 11:11:08.605641 26033 raft_consensus.cc:1261] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260807 11:11:08.621903 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1830) took 1256 ms. Trace:
I20260807 11:11:08.622295 26033 rpcz_store.cc:276] 0807 11:11:07.365214 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:07.623319 (+258105us) service_pool.cc:227] Handling call
0807 11:11:07.950717 (+327398us) spinlock_profiling.cc:243] Waited 2.71 ms on lock 0x7b5800270998. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc461a2b0 00007f3fc46057f9 00007f3fc46052a2 00007f3fbdceb684
0807 11:11:08.604498 (+653781us) spinlock_profiling.cc:243] Waited 21.4 ms on lock 0x7b5800270998. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764b7c 00007f3fbc76473b 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc461a2b0 00007f3fc46057f9 00007f3fc46052a2 00007f3fbdceb684
0807 11:11:08.604935 (+ 437us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:08.621553 (+ 16618us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:08.621829 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":55498880}
I20260807 11:11:08.623972 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:08.610601 25931 raft_consensus.cc:3037] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:08.624495 30184 consensus_queue.cc:1046] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:11:08.627041 29644 leader_election.cc:288] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:08.627795 29644 consensus_queue.cc:1035] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:11:08.630038 29776 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:08.630249 30404 raft_consensus.cc:3032] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 7 LEADER]: Stepping down as leader of term 7
I20260807 11:11:08.630436 26041 raft_consensus.cc:2454] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 7.
I20260807 11:11:08.630839 30404 raft_consensus.cc:729] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:08.632084 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:08.633004 26041 raft_consensus.cc:2454] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:11:08.617252 30112 raft_consensus.cc:2781] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:11:08.633767 30112 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:08.634245 30112 raft_consensus.cc:3037] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:08.634786 26033 raft_consensus.cc:3037] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 6
I20260807 11:11:08.638196 30153 raft_consensus.cc:2781] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:08.634181 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:08.542930 25806 raft_consensus.cc:1261] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260807 11:11:08.640276 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 1.17188e+06us
I20260807 11:11:08.618885 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:08.618497 30344 raft_consensus.cc:3037] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:08.616405 25981 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:08.647792 25806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1845) took 1237 ms. Trace:
I20260807 11:11:08.647966 25797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1855) took 1215 ms. Trace:
I20260807 11:11:08.648655 25797 rpcz_store.cc:276] 0807 11:11:07.434693 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.586817 (+1152124us) service_pool.cc:227] Handling call
0807 11:11:08.614129 (+ 27312us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:08.645459 (+ 31330us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:08.645722 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:08.648312 25806 rpcz_store.cc:276] 0807 11:11:07.406819 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.541768 (+1134949us) service_pool.cc:227] Handling call
0807 11:11:08.542327 (+ 559us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:08.644015 (+101688us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:08.644285 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:08.668084 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:08.645468 30153 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:08.632419 30404 consensus_queue.cc:260] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:08.694494 29798 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:08.695581 29798 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:08.701615 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:08.705562 25799 raft_consensus.cc:1261] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 11:11:08.706430 30247 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:08.691337 26033 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:11:08.708369 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:08.676216 30153 raft_consensus.cc:3037] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
W20260807 11:11:08.711927 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 95693.6us
I20260807 11:11:08.710506 25806 raft_consensus.cc:1261] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260807 11:11:08.721733 25806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 412) took 1069 ms. Trace:
I20260807 11:11:08.722193 25806 rpcz_store.cc:276] 0807 11:11:07.652469 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.653057 (+1000588us) service_pool.cc:227] Handling call
0807 11:11:08.708891 (+ 55834us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:08.721394 (+ 12503us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:08.721664 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:08.630785 29776 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:08.714577 25931 raft_consensus.cc:2454] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 7.
I20260807 11:11:08.675266 30441 raft_consensus.cc:482] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:08.716071 25799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1848) took 1294 ms. Trace:
I20260807 11:11:08.725646 25799 rpcz_store.cc:276] 0807 11:11:07.421938 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.650402 (+1228464us) service_pool.cc:227] Handling call
0807 11:11:08.705144 (+ 54742us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:08.712201 (+ 7057us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:08.715989 (+ 3788us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:08.727317 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:08.728688 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:08.729519 26033 raft_consensus.cc:3037] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:08.736209 30247 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:08.740432 25918 raft_consensus.cc:3037] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:08.742424 26938 leader_election.cc:302] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:08.742703 30094 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:11:08.743316 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 67499us
I20260807 11:11:08.743590 29950 raft_consensus.cc:2781] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:11:08.744241 29950 raft_consensus.cc:686] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 6 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:11:08.745682 30161 raft_consensus.cc:2781] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 11:11:08.746094 30161 raft_consensus.cc:482] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:08.746407 30161 raft_consensus.cc:3037] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:08.749007 30441 raft_consensus.cc:504] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:08.750257 26975 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 491) took 3893 ms (client timeout 2163 ms). Trace:
W20260807 11:11:08.750705 26975 rpcz_store.cc:269] 0807 11:11:04.857099 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:04.869924 (+ 12825us) service_pool.cc:227] Handling call
0807 11:11:07.586129 (+2716205us) spinlock_profiling.cc:243] Waited 4.59 ms on lock 0x7b5800000f98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc46061b0 00007f3fc4605a94 00007f3fbdcedc04 00007f3fbdcedb92
0807 11:11:08.749876 (+1163747us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b5400470754. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:08.750023 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2335111808}
I20260807 11:11:08.751667 30441 leader_election.cc:288] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:08.752219 28964 leader_election.cc:288] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:08.752568 25798 raft_consensus.cc:1261] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260807 11:11:08.753137 30351 raft_consensus.cc:482] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:08.753839 30351 raft_consensus.cc:504] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:08.757639 26936 leader_election.cc:302] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:08.741205 25916 raft_consensus.cc:3037] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:08.737018 30184 raft_consensus.cc:482] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:08.769119 30397 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:08.769507 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1841) took 1342 ms. Trace:
I20260807 11:11:08.769065 30184 raft_consensus.cc:504] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:08.770233 30390 raft_consensus.cc:3032] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:11:08.770747 29991 raft_consensus.cc:2726] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:08.770872 30390 raft_consensus.cc:729] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:08.743625 30094 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:08.753319 25798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1846) took 1346 ms. Trace:
I20260807 11:11:08.769896 30397 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:08.773358 25798 rpcz_store.cc:276] 0807 11:11:07.407272 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.650088 (+1242816us) service_pool.cc:227] Handling call
0807 11:11:08.725001 (+ 74913us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:08.753032 (+ 28031us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:08.753258 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:08.777594 26046 raft_consensus.cc:2454] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:11:08.756533 30351 leader_election.cc:288] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:08.744469 29776 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:11:08.781343 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 1.32592e+06us
I20260807 11:11:08.753007 29680 raft_consensus.cc:2726] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:11:08.787849 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:08.791469 26046 raft_consensus.cc:2379] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 6: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:11:08.791460 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:08.769941 26037 rpcz_store.cc:276] 0807 11:11:07.426906 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.293071 (+866165us) service_pool.cc:227] Handling call
0807 11:11:08.320047 (+ 26976us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:08.328308 (+ 8261us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:08.328342 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:08.331013 (+ 2671us) log.cc:842] Serialized 26 byte log entry
0807 11:11:08.692081 (+361068us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:08.692631 (+ 550us) log.cc:842] Serialized 14 byte log entry
0807 11:11:08.741230 (+ 48599us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:08.769090 (+ 27860us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:08.769156 (+ 66us) raft_consensus.cc:1671] finished
0807 11:11:08.769172 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:08.769312 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27192,"prepare.queue_time_us":585,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":67,"raft.queue_time_us":59904,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":252,"thread_start_us":154102,"threads_started":1,"wal-append.queue_time_us":396790}
I20260807 11:11:08.808998 26033 raft_consensus.cc:2454] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 7.
I20260807 11:11:08.810189 30315 consensus_queue.cc:1035] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260807 11:11:08.816601 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1842) took 1347 ms. Trace:
I20260807 11:11:08.772670 30390 consensus_queue.cc:260] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 3, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:08.883740 26938 leader_election.cc:302] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:08.924239 30415 raft_consensus.cc:686] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 7 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:08.823452 30351 raft_consensus.cc:482] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:09.326886 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:11:09.199126 30492 log.cc:924] Time spent T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.828s user 0.003s sys 0.000s
I20260807 11:11:08.936354 25800 raft_consensus.cc:3032] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 11:11:09.243116 30324 raft_consensus.cc:482] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:09.336474 25800 raft_consensus.cc:729] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
W20260807 11:11:09.261091 30473 log.cc:924] Time spent T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 1.531s user 0.000s sys 0.004s
I20260807 11:11:09.336966 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:09.338275 25801 raft_consensus.cc:2379] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 6: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:11:09.338016 25800 consensus_queue.cc:260] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:09.338546 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 512) took 2563 ms. Trace:
I20260807 11:11:09.339267 25800 raft_consensus.cc:3037] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
W20260807 11:11:09.048000 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 287520us
I20260807 11:11:09.339040 26042 rpcz_store.cc:276] 0807 11:11:06.775290 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:07.199492 (+424202us) service_pool.cc:227] Handling call
0807 11:11:07.249263 (+ 49771us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:07.249659 (+ 396us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:07.249692 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:07.250202 (+ 510us) log.cc:842] Serialized 26 byte log entry
0807 11:11:08.616402 (+1366200us) spinlock_profiling.cc:243] Waited 6.95 ms on lock 0x7b18004181e0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbf35382c 00007f3fbc77e9dc 00007f3fc55b5a39 00007f3fbc79b8bb 00007f3fbc799cbf 00007f3fbc79be22 00007f3fbf3455bb 00007f3fbf34b063 00007f3fbf34b411
0807 11:11:08.616736 (+ 334us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:11:08.628842 (+ 12106us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:08.628964 (+ 122us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:09.259982 (+631018us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:11:09.338120 (+ 78138us) raft_consensus.cc:1671] finished
0807 11:11:09.338147 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:09.338339 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":156199,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":4399,"spinlock_wait_cycles":49762944,"thread_start_us":181774,"threads_started":1,"wal-append.queue_time_us":348851}
I20260807 11:11:09.341602 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:09.342316 26042 raft_consensus.cc:2379] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 6: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:11:09.344169 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:08.808243 25872 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:09.345067 26042 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 9 FOLLOWER]: Advancing to term 10
I20260807 11:11:08.836918 26985 raft_consensus.cc:1226] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 LEADER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 8. Current term is 9. Ops: []
W20260807 11:11:08.937366 25808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1598) took 1586 ms (client timeout 1595 ms). Trace:
I20260807 11:11:08.817094 26039 rpcz_store.cc:276] 0807 11:11:07.435091 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.299964 (+864873us) service_pool.cc:227] Handling call
0807 11:11:08.300307 (+ 343us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:08.300878 (+ 571us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 11:11:08.331965 (+ 31087us) log.cc:842] Serialized 14 byte log entry
0807 11:11:08.714887 (+382922us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:08.757403 (+ 42516us) log.cc:842] Serialized 26 byte log entry
0807 11:11:08.757657 (+ 254us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:08.757698 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:08.757834 (+ 136us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:08.773617 (+ 15783us) raft_consensus.cc:1671] finished
0807 11:11:08.773638 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:08.782200 (+ 8562us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47457,"prepare.queue_time_us":19555,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":16386,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":27337,"thread_start_us":137380,"threads_started":1,"wal-append.queue_time_us":404940}
W20260807 11:11:09.346825 25808 rpcz_store.cc:269] 0807 11:11:07.350364 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:07.392304 (+ 41940us) service_pool.cc:227] Handling call
0807 11:11:08.937017 (+1544713us) spinlock_profiling.cc:243] Waited 7.65 ms on lock 0x7b5400136854. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:08.937172 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17587968}
I20260807 11:11:08.982151 25916 raft_consensus.cc:2454] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:11:09.349769 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:11:08.935138 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 188775us
I20260807 11:11:08.935098 25796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1831) took 5626 ms. Trace:
I20260807 11:11:09.352628 30260 consensus_queue.cc:1035] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:11:09.352072 25796 rpcz_store.cc:276] 0807 11:11:03.308327 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.466866 (+3158539us) service_pool.cc:227] Handling call
0807 11:11:06.470830 (+ 3964us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:06.797240 (+326410us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:08.934603 (+2137363us) spinlock_profiling.cc:243] Waited 326 ms on lock 0x7b5400266454. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:08.934884 (+ 281us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":748871936}
I20260807 11:11:09.353731 30378 consensus_queue.cc:1035] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260807 11:11:08.943181 26978 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 484) took 4077 ms (client timeout 2874 ms). Trace:
I20260807 11:11:09.262315 26986 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50960 (request call id 383) took 1078 ms. Trace:
I20260807 11:11:09.355170 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:09.355913 26986 rpcz_store.cc:276] 0807 11:11:08.188903 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.281489 (+ 92586us) service_pool.cc:227] Handling call
0807 11:11:09.262023 (+980534us) spinlock_profiling.cc:243] Waited 236 ms on lock 0x7b5400760554. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:09.262228 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":542463360}
I20260807 11:11:09.357533 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:09.357385 30378 consensus_queue.cc:1035] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:11:09.179899 29760 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:09.361116 29760 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:08.950449 25918 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:11:08.834601 30278 consensus_queue.cc:1035] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.533s
I20260807 11:11:09.363816 29760 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:09.179808 30266 raft_consensus.cc:482] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:09.365139 30266 raft_consensus.cc:504] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:08.821915 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:09.367566 30266 leader_election.cc:288] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:08.826994 25981 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:08.845961 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:09.369057 30475 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:09.369822 30475 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:09.370858 30232 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.657s user 0.011s sys 0.038s
W20260807 11:11:09.371608 30232 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.658s user 0.011s sys 0.038s
I20260807 11:11:09.372228 30475 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:11:09.372430 30256 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.003s sys 0.018s
W20260807 11:11:09.373050 30256 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.609s user 0.003s sys 0.018s
I20260807 11:11:08.938380 28964 consensus_queue.cc:1035] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.436s
I20260807 11:11:08.899374 30161 raft_consensus.cc:504] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:11:09.355326 26978 rpcz_store.cc:269] 0807 11:11:04.865247 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:05.306338 (+441091us) service_pool.cc:227] Handling call
0807 11:11:08.942728 (+3636390us) spinlock_profiling.cc:243] Waited 1.9 s on lock 0x7b540045bfd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:08.942960 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4370462464}
I20260807 11:11:09.334733 30404 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:09.337023 30324 raft_consensus.cc:504] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:08.896799 26936 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:09.387279 26975 raft_consensus.cc:1226] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260807 11:11:09.350386 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:08.933441 29950 consensus_queue.cc:237] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:09.388638 25796 raft_consensus.cc:2379] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 6: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:11:09.389602 30324 leader_election.cc:288] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:09.361399 30415 consensus_queue.cc:237] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:09.391826 30172 consensus_queue.cc:1046] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:11:09.392058 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:09.368119 26033 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:11:09.393405 25796 raft_consensus.cc:2379] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 7: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:11:09.136229 30153 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:09.003273 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:09.335937 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1845) took 1823 ms. Trace:
I20260807 11:11:09.339494 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:09.393462 26042 raft_consensus.cc:2454] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 10.
I20260807 11:11:08.838128 30171 raft_consensus.cc:2781] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:11:09.399475 26039 raft_consensus.cc:2379] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 7: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
W20260807 11:11:09.415479 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 80962.3us
I20260807 11:11:09.344695 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:09.385675 25916 raft_consensus.cc:1261] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 11:11:09.260964 26984 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50942 (request call id 486) took 1842 ms. Trace:
W20260807 11:11:09.423673 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 646209us
I20260807 11:11:09.343990 30270 raft_consensus.cc:482] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:09.334264 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1844) took 1879 ms. Trace:
W20260807 11:11:09.413626 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 529960us
I20260807 11:11:09.431468 26034 rpcz_store.cc:276] 0807 11:11:07.454624 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.511968 (+1057344us) service_pool.cc:227] Handling call
0807 11:11:08.512414 (+ 446us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:08.519798 (+ 7384us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:08.519835 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:08.521006 (+ 1171us) log.cc:842] Serialized 26 byte log entry
0807 11:11:08.681833 (+160827us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:11:08.681884 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:08.681973 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:08.938013 (+256040us) raft_consensus.cc:1671] finished
0807 11:11:08.938037 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:09.334192 (+396155us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":15425,"raft.run_cpu_time_us":408,"raft.run_wall_time_us":377,"thread_start_us":160308,"threads_started":1,"wal-append.queue_time_us":266038}
I20260807 11:11:08.924680 30391 raft_consensus.cc:3032] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 7 LEADER]: Stepping down as leader of term 7
W20260807 11:11:09.399674 25806 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.663s user 0.005s sys 0.014s
W20260807 11:11:09.438026 25806 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.701s user 0.006s sys 0.016s
W20260807 11:11:09.438335 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 599173us
I20260807 11:11:09.395888 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 520) took 1799 ms. Trace:
I20260807 11:11:09.440939 26033 rpcz_store.cc:276] 0807 11:11:07.596568 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:09.394627 (+1798059us) service_pool.cc:227] Handling call
0807 11:11:09.395192 (+ 565us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:09.395486 (+ 294us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:09.395523 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:09.395571 (+ 48us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:09.395624 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:09.395692 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:09.395816 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:09.243304 24034 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 11:11:09.371675 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 519) took 1814 ms. Trace:
I20260807 11:11:09.048336 25798 raft_consensus.cc:1226] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 10 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260807 11:11:09.437249 26979 raft_consensus.cc:2379] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 6: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
W20260807 11:11:09.384325 25803 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.005s sys 0.004s
I20260807 11:11:09.427529 26984 rpcz_store.cc:276] 0807 11:11:07.418836 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:07.418973 (+ 137us) service_pool.cc:227] Handling call
0807 11:11:07.437155 (+ 18182us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:07.733116 (+295961us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:09.260284 (+1527168us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b5400710554. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:09.260725 (+ 441us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":450684672}
I20260807 11:11:09.444870 30400 raft_consensus.cc:2781] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 11:11:09.348851 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:08.940228 30397 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:11:09.371869 29692 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.664s user 0.003s sys 0.016s
W20260807 11:11:09.099678 30154 consensus_meta.cc:287] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.652s user 0.008s sys 0.003s
I20260807 11:11:09.075049 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:08.885015 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:08.884619 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:09.421998 26041 rpcz_store.cc:276] 0807 11:11:07.512538 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.817291 (+1304753us) service_pool.cc:227] Handling call
0807 11:11:09.335245 (+517954us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:09.335569 (+ 324us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:09.335594 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:09.335677 (+ 83us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:11:09.335718 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:09.335771 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:09.335877 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:10.226581 26037 rpcz_store.cc:276] 0807 11:11:07.557011 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:09.370348 (+1813337us) service_pool.cc:227] Handling call
0807 11:11:09.370788 (+ 440us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:09.371278 (+ 490us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:09.371314 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:09.371375 (+ 61us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:09.371421 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:09.371486 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:09.371604 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:09.556738 26035 raft_consensus.cc:1226] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 10 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
W20260807 11:11:10.228701 25803 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.420s user 0.014s sys 0.017s
I20260807 11:11:09.512130 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:09.445362 29323 raft_consensus.cc:2781] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader election won for term 7
W20260807 11:11:09.578426 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 80126.5us
I20260807 11:11:09.505311 25797 raft_consensus.cc:1226] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:09.612102 25800 raft_consensus.cc:1261] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260807 11:11:10.021761 25931 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.911s user 0.017s sys 0.005s
I20260807 11:11:09.484087 25801 raft_consensus.cc:3037] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:09.683667 26975 raft_consensus.cc:1226] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 LEADER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 7. Current term is 8. Ops: []
W20260807 11:11:09.447047 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 657115us
I20260807 11:11:09.529444 26936 leader_election.cc:302] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:11:09.558075 30112 consensus_meta.cc:287] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.923s user 0.009s sys 0.004s
W20260807 11:11:10.239960 25931 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.129s user 0.021s sys 0.009s
I20260807 11:11:09.541208 29798 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:11:09.545040 26976 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.010s sys 0.008s
W20260807 11:11:10.243669 30530 log.cc:924] Time spent T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.151s user 0.001s sys 0.000s
W20260807 11:11:10.229905 29692 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.522s user 0.012s sys 0.030s
I20260807 11:11:09.660281 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:09.544066 30094 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:11:10.129531 25932 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.005s sys 0.007s
W20260807 11:11:10.269279 25932 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.714s user 0.005s sys 0.009s
I20260807 11:11:09.609655 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 523) took 1625 ms. Trace:
I20260807 11:11:10.095397 30398 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:09.072651 30517 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:11:10.243350 30286 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260807 11:11:09.077951 30184 leader_election.cc:288] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:10.244845 25799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 415) took 2537 ms. Trace:
I20260807 11:11:10.245301 25798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 416) took 2501 ms. Trace:
I20260807 11:11:10.239790 25800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1828) took 7551 ms. Trace:
I20260807 11:11:10.249928 26979 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 494) took 1235 ms. Trace:
I20260807 11:11:09.823480 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:09.437587 30391 raft_consensus.cc:729] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:08.943836 30491 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:09.545024 30124 raft_consensus.cc:3037] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:10.247422 25803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1850) took 2824 ms. Trace:
I20260807 11:11:10.249353 30278 raft_consensus.cc:2781] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260807 11:11:09.617300 30126 consensus_meta.cc:287] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 1.199s user 0.005s sys 0.010s
W20260807 11:11:09.484802 29966 consensus_meta.cc:287] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.926s user 0.008s sys 0.008s
I20260807 11:11:10.090903 29991 raft_consensus.cc:482] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:09.662045 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:11:10.253258 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 813496us
I20260807 11:11:09.561039 30344 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:10.293848 26936 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:11:10.150142 30496 consensus_meta.cc:287] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 1.574s user 0.014s sys 0.003s
I20260807 11:11:10.255757 26980 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 495) took 1252 ms. Trace:
W20260807 11:11:10.023058 30256 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.649s user 0.009s sys 0.011s
I20260807 11:11:10.192938 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:10.257041 26035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 516) took 2822 ms. Trace:
I20260807 11:11:09.430896 30475 raft_consensus.cc:2781] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 11:11:09.445524 30153 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:10.270479 26034 rpcz_store.cc:276] 0807 11:11:07.984703 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:09.458993 (+1474290us) service_pool.cc:227] Handling call
0807 11:11:09.608790 (+149797us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:09.609083 (+ 293us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:09.609123 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:09.609203 (+ 80us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:09.609312 (+ 109us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:09.609391 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:09.609545 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:10.296742 26980 rpcz_store.cc:276] 0807 11:11:09.003066 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:09.074485 (+ 71419us) service_pool.cc:227] Handling call
0807 11:11:10.255659 (+1181174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:09.684193 30161 leader_election.cc:288] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 8 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:09.560706 30462 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:10.253806 26037 raft_consensus.cc:1226] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 6. Current term is 7. Ops: [6.8-6.8]
W20260807 11:11:10.257153 26038 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.671s user 0.009s sys 0.012s
I20260807 11:11:09.559098 29417 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:10.272042 25801 raft_consensus.cc:2454] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 7.
I20260807 11:11:09.545081 30507 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:10.265488 26033 pending_rounds.cc:76] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9: Aborting all ops after (but not including) 2
I20260807 11:11:09.545858 29847 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:10.280181 25800 rpcz_store.cc:276] 0807 11:11:02.688640 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:06.433908 (+3745268us) service_pool.cc:227] Handling call
0807 11:11:06.451910 (+ 18002us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:08.936083 (+2484173us) spinlock_profiling.cc:243] Waited 59.1 ms on lock 0x7b54001d77a0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf654730 00007f3fbf6c6300 00007f3fbf6c7371 00007f3fbf6c3edf 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc
0807 11:11:10.239531 (+1303448us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.239736 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":136026496}
I20260807 11:11:09.553066 25981 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:11:10.253508 26976 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.416s user 0.013s sys 0.020s
I20260807 11:11:10.282445 25798 rpcz_store.cc:276] 0807 11:11:07.731110 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.782135 (+1051025us) service_pool.cc:227] Handling call
0807 11:11:08.839646 (+ 57511us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:10.231559 (+1391913us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.231842 (+ 283us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:09.549220 30171 raft_consensus.cc:686] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:10.279958 25799 rpcz_store.cc:276] 0807 11:11:07.692868 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.740157 (+1047289us) service_pool.cc:227] Handling call
0807 11:11:08.826823 (+ 86666us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:09.066771 (+239948us) raft_consensus.cc:1520] Early marking committed up to index 7
0807 11:11:09.066817 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:09.078702 (+ 11885us) log.cc:842] Serialized 26 byte log entry
0807 11:11:09.383844 (+305142us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:09.383892 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:09.383984 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:09.618412 (+234428us) raft_consensus.cc:1671] finished
0807 11:11:09.618432 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:10.230397 (+611965us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1650432,"thread_start_us":304561,"threads_started":1,"wal-append.queue_time_us":536871}
I20260807 11:11:10.281656 26979 rpcz_store.cc:276] 0807 11:11:09.002564 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:09.002806 (+ 242us) service_pool.cc:227] Handling call
0807 11:11:10.238338 (+1235532us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:10.282963 25926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35276 (request call id 364) took 1457 ms. Trace:
W20260807 11:11:10.260315 29915 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.988s user 0.007s sys 0.004s
I20260807 11:11:10.331478 25926 rpcz_store.cc:276] 0807 11:11:08.822314 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.884496 (+ 62182us) service_pool.cc:227] Handling call
0807 11:11:10.279151 (+1394655us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:10.287247 30517 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:10.288026 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:09.430042 30351 raft_consensus.cc:504] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:10.291414 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1847) took 1657 ms. Trace:
I20260807 11:11:10.290038 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:11:10.260200 29857 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.855s user 0.013s sys 0.011s
I20260807 11:11:10.261433 30500 raft_consensus.cc:2781] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:11:10.295274 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:11:10.306449 30256 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.932s user 0.010s sys 0.014s
I20260807 11:11:10.306725 26035 rpcz_store.cc:276] 0807 11:11:07.434767 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.284162 (+849395us) service_pool.cc:227] Handling call
0807 11:11:09.556131 (+1271969us) spinlock_profiling.cc:243] Waited 1.11 ms on lock 0x7b5800270998. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc461a2b0 00007f3fc46057f9 00007f3fc46052a2 00007f3fbdceb684
0807 11:11:09.556412 (+ 281us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:10.247150 (+690738us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.256973 (+ 9823us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2549632}
I20260807 11:11:10.308864 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:10.295187 30126 raft_consensus.cc:504] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:10.313223 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:10.314545 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:10.315146 25752 leader_election.cc:302] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:10.315918 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:10.251420 30112 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:10.282907 30266 raft_consensus.cc:482] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:11:10.261236 26974 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 492) took 1438 ms (client timeout 1507 ms). Trace:
I20260807 11:11:09.545169 30424 raft_consensus.cc:482] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
W20260807 11:11:10.323364 26038 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.134s user 0.010s sys 0.012s
I20260807 11:11:09.506685 30270 raft_consensus.cc:504] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:10.327055 25801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59118 (request call id 424) took 1510 ms. Trace:
I20260807 11:11:10.362607 25801 rpcz_store.cc:276] 0807 11:11:08.816442 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:09.339188 (+522746us) service_pool.cc:227] Handling call
0807 11:11:10.326997 (+987809us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:10.331856 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1850) took 1695 ms. Trace:
I20260807 11:11:10.336261 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:10.363619 26034 raft_consensus.cc:3037] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:10.285257 25803 rpcz_store.cc:276] 0807 11:11:07.422587 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.579462 (+1156875us) service_pool.cc:227] Handling call
0807 11:11:08.579987 (+ 525us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:08.606687 (+ 26700us) raft_consensus.cc:1520] Early marking committed up to index 7
0807 11:11:08.708312 (+101625us) log.cc:842] Serialized 14 byte log entry
0807 11:11:10.246979 (+1538667us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:10.247054 (+ 75us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:10.247117 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.247202 (+ 85us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:10.247349 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3160,"prepare.queue_time_us":178779,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":61,"raft.queue_time_us":22075,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":15958,"thread_start_us":1502533,"threads_started":2,"wal-append.queue_time_us":1016284}
I20260807 11:11:10.327243 26033 pending_rounds.cc:97] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260807 11:11:10.364492 26033 raft_consensus.cc:2866] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:11:10.257956 30154 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:10.343321 26041 rpcz_store.cc:276] 0807 11:11:08.628279 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.227440 (+1599161us) service_pool.cc:227] Handling call
0807 11:11:10.280383 (+ 52943us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:10.285566 (+ 5183us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:10.285597 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:10.285639 (+ 42us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:10.285670 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.285724 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:10.285833 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:10.350119 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:10.348201 25796 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
W20260807 11:11:10.337275 29915 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.180s user 0.008s sys 0.004s
I20260807 11:11:10.292017 30398 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:11:10.354302 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 61277.1us
I20260807 11:11:10.319726 30475 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:10.376017 26046 raft_consensus.cc:2379] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 6: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
W20260807 11:11:10.353542 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 811188us
W20260807 11:11:10.347424 29857 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.343s user 0.014s sys 0.012s
I20260807 11:11:10.360956 30421 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:10.361169 30507 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:10.361545 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:10.363435 26037 rpcz_store.cc:276] 0807 11:11:08.629555 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.247074 (+1617519us) service_pool.cc:227] Handling call
0807 11:11:10.251878 (+ 4804us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:10.324079 (+ 72201us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.324394 (+ 315us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:10.358834 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:10.362520 25806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 414) took 2688 ms. Trace:
I20260807 11:11:10.356427 25808 raft_consensus.cc:3037] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:10.409895 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:10.414211 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:10.417364 25806 rpcz_store.cc:276] 0807 11:11:07.674528 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.734785 (+1060257us) service_pool.cc:227] Handling call
0807 11:11:08.735328 (+ 543us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:08.735863 (+ 535us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:08.735891 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:08.736368 (+ 477us) log.cc:842] Serialized 26 byte log entry
0807 11:11:09.438414 (+702046us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:11:09.438451 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:09.438512 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:10.362203 (+923691us) raft_consensus.cc:1671] finished
0807 11:11:10.362247 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:10.362441 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":17223,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":243,"thread_start_us":701549,"threads_started":1,"wal-append.queue_time_us":1379721}
I20260807 11:11:10.411942 25752 leader_election.cc:302] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:10.412974 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:10.319857 29991 raft_consensus.cc:504] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:10.337432 30344 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:10.287276 30491 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:10.422578 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1840) took 3014 ms. Trace:
I20260807 11:11:10.429827 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:11:10.416446 30232 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.856s user 0.055s sys 0.099s
W20260807 11:11:10.435356 30531 log.cc:924] Time spent T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
I20260807 11:11:10.433056 26038 rpcz_store.cc:276] 0807 11:11:07.408488 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:07.877951 (+469463us) service_pool.cc:227] Handling call
0807 11:11:08.285506 (+407555us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:11:08.285861 (+ 355us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:08.285890 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 11:11:08.286527 (+ 637us) log.cc:842] Serialized 52 byte log entry
0807 11:11:08.810844 (+524317us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:08.921481 (+110637us) log.cc:842] Serialized 14 byte log entry
0807 11:11:08.961203 (+ 39722us) log.cc:842] Serialized 14 byte log entry
0807 11:11:09.189049 (+227846us) spinlock_profiling.cc:243] Waited 1.21 ms on lock 0x7b18003e0380. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbc77e345 00007f3fc55b5a39 00007f3fbc79b8bb 00007f3fbc799cbf 00007f3fbc799155 00007f3fbf6bc3d0 00007f3fbf6ce576 00007f3fbf6df0d7 00007f3fbf6df058
0807 11:11:10.359011 (+1169962us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.410421 (+ 51410us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:10.422259 (+ 11838us) raft_consensus.cc:1671] finished
0807 11:11:10.422291 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:10.422493 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3513,"prepare.queue_time_us":105240,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":106,"raft.queue_time_us":41985,"raft.run_cpu_time_us":477,"raft.run_wall_time_us":46556,"spinlock_wait_cycles":2797312,"thread_start_us":1565314,"threads_started":2,"wal-append.queue_time_us":436677,"wal-append.run_cpu_time_us":4632,"wal-append.run_wall_time_us":1235946}
I20260807 11:11:10.360072 30153 raft_consensus.cc:2781] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:10.364883 30500 raft_consensus.cc:686] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:10.409633 30475 raft_consensus.cc:3037] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:10.437579 26037 raft_consensus.cc:1261] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260807 11:11:10.369911 30397 raft_consensus.cc:2781] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 11:11:10.359833 29847 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:11:10.356524 26974 rpcz_store.cc:269] 0807 11:11:08.826136 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.883979 (+ 57843us) service_pool.cc:227] Handling call
0807 11:11:10.261172 (+1377193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:10.444607 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:10.445463 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:10.450886 25916 raft_consensus.cc:1261] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 11:11:10.448037 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 528) took 1791 ms. Trace:
W20260807 11:11:10.440282 30232 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.880s user 0.056s sys 0.100s
I20260807 11:11:10.447490 25917 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 8.
I20260807 11:11:10.452973 26037 rpcz_store.cc:276] 0807 11:11:08.665568 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.418900 (+1753332us) service_pool.cc:227] Handling call
0807 11:11:10.431269 (+ 12369us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:10.442795 (+ 11526us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.442992 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:10.457821 25796 raft_consensus.cc:1261] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 11:11:10.458532 25796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1857) took 2667 ms. Trace:
I20260807 11:11:10.459303 25801 raft_consensus.cc:1226] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 8 LEADER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260807 11:11:10.454160 25919 raft_consensus.cc:2379] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 7: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:11:10.460006 25801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1859) took 2637 ms. Trace:
I20260807 11:11:10.435905 29851 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:10.460587 25801 rpcz_store.cc:276] 0807 11:11:07.822614 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.428741 (+2606127us) service_pool.cc:227] Handling call
0807 11:11:10.458970 (+ 30229us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:10.459763 (+ 793us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.459958 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1280}
I20260807 11:11:10.455869 25798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 417) took 2705 ms. Trace:
I20260807 11:11:10.458931 25796 rpcz_store.cc:276] 0807 11:11:07.790662 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.411264 (+2620602us) service_pool.cc:227] Handling call
0807 11:11:10.449850 (+ 38586us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:10.458217 (+ 8367us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.458471 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:10.443079 29915 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:10.437189 26976 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50942 (request call id 490) took 1722 ms. Trace:
I20260807 11:11:10.465157 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1856) took 1812 ms. Trace:
I20260807 11:11:10.425343 30266 raft_consensus.cc:504] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:10.465499 25798 rpcz_store.cc:276] 0807 11:11:07.750585 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.343609 (+2593024us) service_pool.cc:227] Handling call
0807 11:11:10.424680 (+ 81071us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:10.455546 (+ 30866us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:10.455573 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:10.455614 (+ 41us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:10.455649 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.455705 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:10.455812 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:10.359372 30462 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:10.442533 30153 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:10.465957 25806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1858) took 2669 ms. Trace:
I20260807 11:11:10.475193 25806 rpcz_store.cc:276] 0807 11:11:07.796664 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.427252 (+2630588us) service_pool.cc:227] Handling call
0807 11:11:10.456425 (+ 29173us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:10.456591 (+ 166us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:10.456615 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:10.465589 (+ 8974us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:10.465658 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.465753 (+ 95us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:10.465887 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:10.351712 29417 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:10.469678 26038 rpcz_store.cc:276] 0807 11:11:08.666144 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.446088 (+1779944us) service_pool.cc:227] Handling call
0807 11:11:10.464635 (+ 18547us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:10.464841 (+ 206us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:10.464864 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:10.464912 (+ 48us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:10.464953 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.465002 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:10.465104 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:10.471917 30517 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:10.410234 30424 raft_consensus.cc:504] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:10.459601 29966 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:10.425998 30421 raft_consensus.cc:504] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:10.481495 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:10.476327 30153 raft_consensus.cc:3037] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:10.490515 25801 raft_consensus.cc:2454] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 7.
I20260807 11:11:10.470538 30286 raft_consensus.cc:2781] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:10.490768 25799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 418) took 1742 ms. Trace:
I20260807 11:11:10.494634 25799 rpcz_store.cc:276] 0807 11:11:08.748664 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.441841 (+1693177us) service_pool.cc:227] Handling call
0807 11:11:10.460642 (+ 18801us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:10.490408 (+ 29766us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:10.490439 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:10.490480 (+ 41us) raft_consensus.cc:1631] Marking committed up to 8
0807 11:11:10.490518 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.490573 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:10.490705 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:10.470340 30397 raft_consensus.cc:686] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 7 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:11:10.503105 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:10.488194 30112 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:10.513391 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:10.486557 29851 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:10.516450 25806 raft_consensus.cc:1226] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260807 11:11:10.498909 30286 raft_consensus.cc:482] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:10.522274 25806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 421) took 1731 ms. Trace:
I20260807 11:11:10.522657 25806 rpcz_store.cc:276] 0807 11:11:08.790909 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.476640 (+1685731us) service_pool.cc:227] Handling call
0807 11:11:10.496926 (+ 20286us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:10.521986 (+ 25060us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.522217 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:10.519327 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 529) took 1750 ms. Trace:
I20260807 11:11:10.470116 30171 consensus_queue.cc:237] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:11:10.536095 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 309641us
I20260807 11:11:10.515012 29798 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:10.523414 26038 rpcz_store.cc:276] 0807 11:11:08.768744 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.479146 (+1710402us) service_pool.cc:227] Handling call
0807 11:11:10.499392 (+ 20246us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:10.511439 (+ 12047us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:10.511457 (+ 18us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:10.511487 (+ 30us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:10.519037 (+ 7550us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.519115 (+ 78us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:10.519221 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:11:10.533984 30524 log.cc:924] Time spent T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.175s user 0.002s sys 0.000s
I20260807 11:11:10.524263 30286 raft_consensus.cc:3037] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:10.544562 25806 raft_consensus.cc:1226] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 5. Current term is 6. Ops: []
I20260807 11:11:10.486809 29915 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:10.529464 30126 leader_election.cc:288] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:10.521433 29966 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:10.546916 30507 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:10.459738 30391 consensus_queue.cc:260] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:10.534886 30154 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:10.554072 26038 raft_consensus.cc:1261] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 11:11:10.557747 25981 leader_election.cc:302] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:10.562108 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1936) took 1626 ms. Trace:
I20260807 11:11:10.564399 25981 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:10.562813 26038 rpcz_store.cc:276] 0807 11:11:08.935619 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.539003 (+1603384us) service_pool.cc:227] Handling call
0807 11:11:10.553533 (+ 14530us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:10.557330 (+ 3797us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.562037 (+ 4707us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:10.595728 29323 raft_consensus.cc:686] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:10.596356 30552 raft_consensus.cc:2781] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:11:10.597272 30552 raft_consensus.cc:686] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:10.597051 29323 consensus_queue.cc:237] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:10.598944 30552 consensus_queue.cc:237] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:10.558996 30094 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:10.552946 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:10.557725 30391 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:10.563397 30270 leader_election.cc:288] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:11:10.630064 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 65788.8us
I20260807 11:11:10.554664 30283 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:11:10.606681 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 194891us
I20260807 11:11:10.556066 30491 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:11:10.609628 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 315854us
I20260807 11:11:10.610395 26038 raft_consensus.cc:1261] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260807 11:11:10.637403 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1937) took 1701 ms. Trace:
I20260807 11:11:10.637868 26038 rpcz_store.cc:276] 0807 11:11:08.936002 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.586902 (+1650900us) service_pool.cc:227] Handling call
0807 11:11:10.609910 (+ 23008us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:10.637026 (+ 27116us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.637323 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:10.557767 30344 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:10.564657 30112 raft_consensus.cc:2781] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:10.607816 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1849) took 1978 ms. Trace:
I20260807 11:11:10.571381 30351 leader_election.cc:288] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:10.590510 30398 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:10.539592 30500 consensus_queue.cc:237] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:10.639324 26033 rpcz_store.cc:276] 0807 11:11:08.629100 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.230922 (+1601822us) service_pool.cc:227] Handling call
0807 11:11:10.265158 (+ 34236us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:11:10.429170 (+164012us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:10.429231 (+ 61us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 11:11:10.437159 (+ 7928us) log.cc:842] Serialized 52 byte log entry
0807 11:11:10.591402 (+154243us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:11:10.591461 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.591551 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:10.607565 (+ 16014us) raft_consensus.cc:1671] finished
0807 11:11:10.607589 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:10.607748 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2652,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":3044,"thread_start_us":131228,"threads_started":1,"wal-append.queue_time_us":147451}
I20260807 11:11:10.665802 25806 raft_consensus.cc:1261] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 11:11:10.668409 26038 raft_consensus.cc:3032] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 5 LEADER]: Stepping down as leader of term 5
W20260807 11:11:10.668690 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 414788us
I20260807 11:11:10.671973 26038 raft_consensus.cc:729] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:11:10.635216 30400 raft_consensus.cc:686] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:10.674937 26987 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50942 (request call id 493) took 1516 ms. Trace:
I20260807 11:11:10.607259 29991 leader_election.cc:288] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:10.678586 26987 rpcz_store.cc:276] 0807 11:11:09.002050 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:09.002345 (+ 295us) service_pool.cc:227] Handling call
0807 11:11:09.002869 (+ 524us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:09.003477 (+ 608us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:09.003508 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:09.534271 (+530763us) log.cc:842] Serialized 26 byte log entry
0807 11:11:10.075106 (+540835us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:10.075229 (+ 123us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.075393 (+ 164us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:10.518619 (+443226us) raft_consensus.cc:1671] finished
0807 11:11:10.518640 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:10.518767 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":360,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":100,"thread_start_us":333511,"threads_started":1,"wal-append.queue_time_us":850458}
I20260807 11:11:10.662067 30126 raft_consensus.cc:482] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:10.588174 29798 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:10.709630 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:10.658478 30112 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:10.677565 26038 consensus_queue.cc:260] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:10.707381 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 45439.7us
I20260807 11:11:10.706303 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 363) took 1919 ms. Trace:
I20260807 11:11:10.657897 30266 leader_election.cc:288] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:10.635885 29847 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:10.675925 29851 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:11:10.739847 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 62233.2us
W20260807 11:11:10.744496 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 56803.1us
I20260807 11:11:10.687496 29417 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:10.676043 30462 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:10.665222 30094 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:10.716490 25750 connection.cc:232] RPC call timeout handler was delayed by -2.7280052718706429s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19730000000000003s for a response
I20260807 11:11:10.758775 25800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1860) took 2917 ms. Trace:
I20260807 11:11:10.718839 30112 raft_consensus.cc:3037] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:10.469328 26976 rpcz_store.cc:276] 0807 11:11:08.714374 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.736835 (+ 22461us) service_pool.cc:227] Handling call
0807 11:11:08.742483 (+ 5648us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:08.742970 (+ 487us) raft_consensus.cc:1520] Early marking committed up to index 7
0807 11:11:08.785684 (+ 42714us) log.cc:842] Serialized 14 byte log entry
0807 11:11:10.359540 (+1573856us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:10.370194 (+ 10654us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:10.412395 (+ 42201us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.421663 (+ 9268us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:10.432530 (+ 10867us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35899,"prepare.queue_time_us":43592,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":42,"thread_start_us":1499515,"threads_started":1,"wal-append.queue_time_us":1477148}
I20260807 11:11:10.772192 30517 consensus_queue.cc:1035] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:11:10.774718 30557 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
W20260807 11:11:10.775733 30537 log.cc:924] Time spent T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.250s user 0.000s sys 0.001s
I20260807 11:11:10.775471 30557 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:10.747300 25872 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.696s (SENT) [suppressed 4 similar messages]
I20260807 11:11:10.761792 26038 raft_consensus.cc:3037] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
W20260807 11:11:10.762705 25750 connection.cc:232] RPC call timeout handler was delayed by -2.6281052720732987s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20840000000000003s for a response
I20260807 11:11:10.762784 25800 rpcz_store.cc:276] 0807 11:11:07.836496 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.433519 (+2597023us) service_pool.cc:227] Handling call
0807 11:11:10.455902 (+ 22383us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:10.518237 (+ 62335us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:10.518266 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:10.518311 (+ 45us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:10.518359 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.753265 (+234906us) spinlock_profiling.cc:243] Waited 62.1 ms on lock 0x7b54002d3cd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:10.753334 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:10.753511 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":143447808}
W20260807 11:11:10.778772 25750 connection.cc:232] RPC call timeout handler was delayed by -2.5570052720140666s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2163s for a response
I20260807 11:11:10.715027 25932 rpcz_store.cc:276] 0807 11:11:08.790010 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.807063 (+ 17053us) service_pool.cc:227] Handling call
0807 11:11:09.429217 (+622154us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:09.555105 (+125888us) raft_consensus.cc:1520] Early marking committed up to index 7
0807 11:11:10.269892 (+714787us) log.cc:842] Serialized 14 byte log entry
0807 11:11:10.705832 (+435940us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:10.705909 (+ 77us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:10.705979 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.706065 (+ 86us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:10.706218 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29574,"prepare.queue_time_us":57991,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":46245,"thread_start_us":1104194,"threads_started":2}
W20260807 11:11:10.779117 25750 connection.cc:232] RPC call timeout handler was delayed by -1.9990052720531821s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2783s for a response
I20260807 11:11:10.665350 30397 consensus_queue.cc:237] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:10.766263 25919 raft_consensus.cc:1261] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260807 11:11:10.766115 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:10.766691 30232 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:10.739008 30256 consensus_queue.cc:1046] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20260807 11:11:10.768976 30533 raft_consensus.cc:2781] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 11:11:10.792613 30533 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 9 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:10.793119 30533 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 9 FOLLOWER]: Advancing to term 10
W20260807 11:11:10.795260 30124 consensus_meta.cc:287] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.502s user 0.008s sys 0.000s
W20260807 11:11:10.769428 30534 log.cc:924] Time spent T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.245s user 0.000s sys 0.001s
I20260807 11:11:10.784228 25981 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:10.800418 30351 raft_consensus.cc:2781] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:10.675089 30283 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:10.769261 25869 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:10.769694 30544 consensus_queue.cc:1035] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.105s
I20260807 11:11:10.769873 29847 consensus_queue.cc:1035] T 955db4a12dc04bfe8124047fc46823d3 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:11:10.770565 30184 consensus_queue.cc:1035] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260807 11:11:10.809902 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 131955us
I20260807 11:11:10.807228 25925 raft_consensus.cc:2454] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 7.
I20260807 11:11:10.802353 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:10.759161 30424 leader_election.cc:288] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:11:10.814141 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 53199.7us
W20260807 11:11:10.822495 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 53315us
I20260807 11:11:10.752663 30126 raft_consensus.cc:504] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:10.828006 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:10.859344 25925 raft_consensus.cc:3032] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 8 LEADER]: Stepping down as leader of term 8
I20260807 11:11:10.862275 25925 raft_consensus.cc:729] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:10.800840 30351 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:10.825821 25750 leader_election.cc:302] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:10.823757 25752 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:10.861745 25869 leader_election.cc:302] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:10.796219 30286 raft_consensus.cc:504] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:10.877609 26975 raft_consensus.cc:1261] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260807 11:11:10.806169 29966 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:10.805563 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:10.882400 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:11:10.882447 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 58781.3us
W20260807 11:11:10.883752 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 136496us
W20260807 11:11:10.883991 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 74951.5us
I20260807 11:11:10.825855 30256 raft_consensus.cc:2726] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260807 11:11:10.906060 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 99394us
I20260807 11:11:10.910547 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:10.912421 25919 raft_consensus.cc:3032] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Stepping down as leader of term 6
I20260807 11:11:10.914947 25919 raft_consensus.cc:729] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:10.874310 30351 raft_consensus.cc:3037] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:10.801795 30232 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:10.925760 25921 raft_consensus.cc:2454] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 6.
I20260807 11:11:10.927274 30390 raft_consensus.cc:3037] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
W20260807 11:11:10.927748 25921 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1626) took 3846 ms (client timeout 2104 ms). Trace:
I20260807 11:11:10.930517 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:11:10.928193 25921 rpcz_store.cc:269] 0807 11:11:07.081080 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:07.179934 (+ 98854us) service_pool.cc:227] Handling call
0807 11:11:10.927510 (+3747576us) spinlock_profiling.cc:243] Waited 3.6 s on lock 0x7b54001806d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:10.927646 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8281417472}
I20260807 11:11:10.939386 25932 raft_consensus.cc:2379] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 6: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:11:10.913456 25925 consensus_queue.cc:260] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:10.822073 29991 consensus_queue.cc:1046] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.092s
I20260807 11:11:10.902892 29966 raft_consensus.cc:2781] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 10 FOLLOWER]: Leader election won for term 10
I20260807 11:11:10.934214 26035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1858) took 2176 ms. Trace:
W20260807 11:11:10.957965 30532 log.cc:924] Time spent T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.334s user 0.001s sys 0.001s
I20260807 11:11:10.958822 25750 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:10.959190 30551 raft_consensus.cc:482] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:10.959767 30551 raft_consensus.cc:504] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:10.960780 25931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 362) took 2240 ms. Trace:
I20260807 11:11:10.928974 30320 raft_consensus.cc:482] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:11:10.966454 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 88781.8us
I20260807 11:11:10.947263 30154 raft_consensus.cc:2726] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:11:10.961153 25931 rpcz_store.cc:276] 0807 11:11:08.773274 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.778834 (+ 5560us) service_pool.cc:227] Handling call
0807 11:11:08.808600 (+ 29766us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:08.946683 (+138083us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:08.946732 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:09.062248 (+115516us) log.cc:842] Serialized 26 byte log entry
0807 11:11:10.240899 (+1178651us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:10.240935 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.241033 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:10.960534 (+719501us) raft_consensus.cc:1671] finished
0807 11:11:10.960552 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:10.960677 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":73448,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":225,"thread_start_us":1129881,"threads_started":1,"wal-append.queue_time_us":1401617}
I20260807 11:11:10.899040 30398 consensus_queue.cc:1035] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.056s
I20260807 11:11:10.961107 30266 consensus_queue.cc:1035] T b1765495856a47f09ddbb303f76c635f P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260807 11:11:10.985248 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 162097us
I20260807 11:11:10.965584 30256 raft_consensus.cc:2726] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 11:11:10.987959 25925 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Advancing to term 9
I20260807 11:11:10.997354 25750 leader_election.cc:302] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:10.971402 25919 consensus_queue.cc:260] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:10.961489 30286 leader_election.cc:288] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:10.968058 29915 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:11.005558 25931 raft_consensus.cc:1226] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 7. Current term is 8. Ops: []
I20260807 11:11:10.975772 29966 raft_consensus.cc:686] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [term 10 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
W20260807 11:11:11.001816 30555 log.cc:924] Time spent T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.217s user 0.000s sys 0.001s
I20260807 11:11:11.023562 26936 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:10.982132 30557 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:10.887306 30400 consensus_queue.cc:237] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:11.033965 26033 raft_consensus.cc:1226] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260807 11:11:11.034874 25919 raft_consensus.cc:3037] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
W20260807 11:11:11.037555 26046 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59312 (request call id 1859) took 2272 ms (client timeout 1696 ms). Trace:
W20260807 11:11:11.038106 26046 rpcz_store.cc:269] 0807 11:11:08.757899 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.818141 (+ 60242us) service_pool.cc:227] Handling call
0807 11:11:11.030565 (+2212424us) spinlock_profiling.cc:243] Waited 947 ms on lock 0x7b5400229f54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:11.030726 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2178441088}
W20260807 11:11:11.041343 30563 log.cc:924] Time spent T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.185s user 0.000s sys 0.001s
W20260807 11:11:11.041934 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 44699.3us
I20260807 11:11:11.012732 30551 leader_election.cc:288] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:11.025271 30286 raft_consensus.cc:2726] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 11:11:10.978734 30320 raft_consensus.cc:504] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:11.077160 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 46346.2us
I20260807 11:11:11.080055 25800 raft_consensus.cc:1261] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260807 11:11:11.054630 30528 log.cc:924] Time spent T 76dbf7480b434c0ba6a902c817cf5455 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.351s user 0.000s sys 0.001s
I20260807 11:11:11.010679 30154 consensus_queue.cc:1035] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.061s
I20260807 11:11:11.099946 25869 leader_election.cc:302] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:11.102421 25869 leader_election.cc:302] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:11.070626 29857 raft_consensus.cc:2726] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:11.106196 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:11.116104 25925 raft_consensus.cc:1261] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260807 11:11:11.074970 30094 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:11:11.113963 30574 log.cc:924] Time spent T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.157s user 0.001s sys 0.000s
W20260807 11:11:11.120971 25808 consensus_meta.cc:287] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.691s user 0.008s sys 0.004s
I20260807 11:11:11.110417 29798 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:11.037649 29915 consensus_queue.cc:1035] T 3395e44a980a4ac2bc07304ae2525b35 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.071s
I20260807 11:11:11.121464 25808 raft_consensus.cc:2454] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:11:11.099249 30126 leader_election.cc:288] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:11.098960 30283 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:11:11.130872 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 87884.4us
I20260807 11:11:11.134068 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1846) took 3343 ms. Trace:
I20260807 11:11:11.076084 30556 consensus_queue.cc:1046] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.201s
I20260807 11:11:11.144840 25917 raft_consensus.cc:1226] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260807 11:11:11.145035 26034 consensus_meta.cc:287] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.766s user 0.011s sys 0.002s
I20260807 11:11:11.145445 26034 raft_consensus.cc:2454] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 8.
I20260807 11:11:11.146064 26041 raft_consensus.cc:1226] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260807 11:11:11.146433 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:11.147082 26034 raft_consensus.cc:2454] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:11:11.114351 30424 raft_consensus.cc:2726] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:11:11.148609 26938 leader_election.cc:302] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:11.142333 26042 rpcz_store.cc:276] 0807 11:11:07.790980 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:09.504863 (+1713883us) service_pool.cc:227] Handling call
0807 11:11:09.740536 (+235673us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:10.091826 (+351290us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:10.091905 (+ 79us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:10.250959 (+159054us) log.cc:842] Serialized 26 byte log entry
0807 11:11:10.594524 (+343565us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:10.594574 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.594662 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:11.133846 (+539184us) raft_consensus.cc:1671] finished
0807 11:11:11.133869 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:11.134003 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":62297,"raft.queue_time_us":336196,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":101,"thread_start_us":271921,"threads_started":1,"wal-append.queue_time_us":305827}
I20260807 11:11:11.123126 30124 raft_consensus.cc:2726] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:11.161460 25750 leader_election.cc:302] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:11:11.162904 30550 log.cc:924] Time spent T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.198s user 0.001s sys 0.000s
I20260807 11:11:11.166625 25799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 427) took 2196 ms. Trace:
I20260807 11:11:11.166764 30577 raft_consensus.cc:2781] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 11:11:11.167129 25799 rpcz_store.cc:276] 0807 11:11:08.970761 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.501460 (+1530699us) service_pool.cc:227] Handling call
0807 11:11:10.521070 (+ 19610us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:10.546266 (+ 25196us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:10.546315 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:10.579854 (+ 33539us) log.cc:842] Serialized 26 byte log entry
0807 11:11:10.810819 (+230965us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:11:10.810877 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.810977 (+ 100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:11.166308 (+355331us) raft_consensus.cc:1671] finished
0807 11:11:11.166328 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:11.166519 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":54397,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":2535,"thread_start_us":121552,"threads_started":1,"wal-append.queue_time_us":441077}
I20260807 11:11:11.162992 26046 raft_consensus.cc:2396] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260807 11:11:11.054947 30562 consensus_queue.cc:1035] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.162s
I20260807 11:11:11.171875 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:11.174145 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:11.174882 26034 raft_consensus.cc:2454] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 7.
W20260807 11:11:11.129299 30566 consensus_queue.cc:1159] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b22d69db45194908ad3d345eba9badd9" responder_term: 7 status { last_received { term: 5 index: 7 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 6. Current term is 7. Ops: [6.8-6.8]" } } last_received_current_leader { term: 0 index: 0 } }
I20260807 11:11:11.178840 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:11:11.181051 30584 log.cc:924] Time spent T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.176s user 0.000s sys 0.001s
W20260807 11:11:11.182919 30515 log.cc:924] Time spent T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 1.739s user 0.013s sys 0.008s
I20260807 11:11:11.184734 26036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 515) took 3750 ms. Trace:
I20260807 11:11:11.184923 25798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 420) took 2401 ms. Trace:
W20260807 11:11:11.186223 30595 log.cc:924] Time spent T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260807 11:11:11.185379 25798 rpcz_store.cc:276] 0807 11:11:08.783751 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.470122 (+1686371us) service_pool.cc:227] Handling call
0807 11:11:10.487536 (+ 17414us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:10.496534 (+ 8998us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:10.496583 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:10.530732 (+ 34149us) log.cc:842] Serialized 26 byte log entry
0807 11:11:10.824373 (+293641us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:10.824513 (+ 140us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.824668 (+ 155us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:11.184689 (+360021us) raft_consensus.cc:1671] finished
0807 11:11:11.184708 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:11.184866 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":383,"raft.queue_time_us":4193,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":12996,"thread_start_us":255409,"threads_started":1,"wal-append.queue_time_us":573568}
I20260807 11:11:11.156277 30094 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:11.188228 30094 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:11.189659 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1860) took 2431 ms. Trace:
I20260807 11:11:11.190973 30094 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:11.190150 26037 rpcz_store.cc:276] 0807 11:11:08.758189 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.461828 (+1703639us) service_pool.cc:227] Handling call
0807 11:11:10.483258 (+ 21430us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:10.500932 (+ 17674us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:10.500963 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:10.516007 (+ 15044us) log.cc:842] Serialized 26 byte log entry
0807 11:11:10.826159 (+310152us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:10.826433 (+ 274us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.826503 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:11.189393 (+362890us) raft_consensus.cc:1671] finished
0807 11:11:11.189418 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:11.189589 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9498,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":12502,"thread_start_us":275118,"threads_started":1,"wal-append.queue_time_us":626403}
I20260807 11:11:11.192201 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:11.196386 30256 consensus_queue.cc:1046] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:11:11.197790 30256 consensus_queue.cc:1035] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 11:11:11.199510 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:11.199560 30126 raft_consensus.cc:3032] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 5 LEADER]: Stepping down as leader of term 5
I20260807 11:11:11.200814 30126 raft_consensus.cc:729] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:11.201637 30126 consensus_queue.cc:260] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:11.202821 30126 raft_consensus.cc:3037] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:11.203178 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:11.203896 26046 raft_consensus.cc:2454] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:11:11.204824 30577 raft_consensus.cc:686] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:11.198659 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:11.205909 30577 consensus_queue.cc:237] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:11.206472 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 522) took 3476 ms. Trace:
I20260807 11:11:11.153394 29966 consensus_queue.cc:237] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:11.206930 26039 rpcz_store.cc:276] 0807 11:11:07.729855 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:09.483423 (+1753568us) service_pool.cc:227] Handling call
0807 11:11:09.605364 (+121941us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:10.412568 (+807204us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:10.412597 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:10.439737 (+ 27140us) log.cc:842] Serialized 26 byte log entry
0807 11:11:10.604751 (+165014us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:11:10.604813 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:11.206054 (+601241us) spinlock_profiling.cc:243] Waited 754 ms on lock 0x7b54003732d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:11.206118 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:11.206174 (+ 56us) raft_consensus.cc:1671] finished
0807 11:11:11.206186 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:11.206411 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":36550,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":16195,"spinlock_wait_cycles":1733864192,"thread_start_us":141484,"threads_started":1,"wal-append.queue_time_us":230328}
I20260807 11:11:11.209338 30598 raft_consensus.cc:2726] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260807 11:11:11.210103 30587 log.cc:924] Time spent T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.239s user 0.002s sys 0.000s
W20260807 11:11:11.211776 30522 log.cc:924] Time spent T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 1.607s user 0.013s sys 0.009s
I20260807 11:11:11.213639 25797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 413) took 3560 ms. Trace:
I20260807 11:11:11.214051 25797 rpcz_store.cc:276] 0807 11:11:07.652833 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.707965 (+1055132us) service_pool.cc:227] Handling call
0807 11:11:08.786187 (+ 78222us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:10.251201 (+1465014us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:11.213403 (+962202us) spinlock_profiling.cc:243] Waited 660 ms on lock 0x7b5400266e54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:11.213576 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1517657472}
I20260807 11:11:11.159584 30232 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:11.215240 30533 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:11.216305 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1939) took 2279 ms. Trace:
I20260807 11:11:11.172526 26042 raft_consensus.cc:3037] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:11.216657 25801 raft_consensus.cc:1226] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [term 7 LEADER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:11.216743 26033 rpcz_store.cc:276] 0807 11:11:08.936790 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.666026 (+1729236us) service_pool.cc:227] Handling call
0807 11:11:11.030705 (+364679us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:11.039225 (+ 8520us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:11.216110 (+176885us) spinlock_profiling.cc:243] Waited 341 ms on lock 0x7b5400229f50. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:11.216241 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":783310208}
I20260807 11:11:11.218137 25916 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1812) took 1664 ms. Trace:
I20260807 11:11:11.220757 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:11.182677 30124 consensus_queue.cc:1035] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:11:11.262681 26037 raft_consensus.cc:2379] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 10: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
W20260807 11:11:11.216662 25786 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1584) took 3911 ms (client timeout 1934 ms). Trace:
I20260807 11:11:11.221719 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:11.223402 25916 rpcz_store.cc:276] 0807 11:11:09.554070 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:09.636315 (+ 82245us) service_pool.cc:227] Handling call
0807 11:11:09.834095 (+197780us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:10.452048 (+617953us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:11.217913 (+765865us) spinlock_profiling.cc:243] Waited 612 ms on lock 0x7b5400184a54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:11.218078 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1408707968}
I20260807 11:11:10.954113 26035 rpcz_store.cc:276] 0807 11:11:08.757559 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.449146 (+1691587us) service_pool.cc:227] Handling call
0807 11:11:10.458157 (+ 9011us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:10.458396 (+ 239us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 11:11:10.632057 (+173661us) log.cc:842] Serialized 14 byte log entry
0807 11:11:10.933739 (+301682us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:10.933816 (+ 77us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:10.933892 (+ 76us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.933984 (+ 92us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:10.934139 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":342421,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":104,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":14036,"thread_start_us":418268,"threads_started":2}
I20260807 11:11:11.262468 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:11.247818 26938 leader_election.cc:302] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:11.230062 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:11:11.266620 25869 connection.cc:232] RPC call timeout handler was delayed by -1.3403758748900145s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1507s for a response
W20260807 11:11:11.267549 25869 connection.cc:232] RPC call timeout handler was delayed by -1.0775758749805391s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1799s for a response
I20260807 11:11:11.249238 26042 raft_consensus.cc:2454] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:11:11.269563 26039 raft_consensus.cc:2396] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 7 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 11:11:11.252588 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:11.215271 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:11:11.263451 25786 rpcz_store.cc:269] 0807 11:11:07.305560 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:07.313870 (+ 8310us) service_pool.cc:227] Handling call
0807 11:11:11.216305 (+3902435us) spinlock_profiling.cc:243] Waited 3.37 s on lock 0x7b54000e4cd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:11.216466 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7756143360}
I20260807 11:11:11.271353 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:11.273555 25925 raft_consensus.cc:2454] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 6.
I20260807 11:11:11.274446 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:11.274647 25920 raft_consensus.cc:1226] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260807 11:11:11.275098 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:11.275764 25797 raft_consensus.cc:2454] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
W20260807 11:11:11.220569 30592 log.cc:924] Time spent T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.115s user 0.001s sys 0.000s
I20260807 11:11:11.276626 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:11:11.584192 26038 consensus_meta.cc:287] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.806s user 0.009s sys 0.000s
I20260807 11:11:11.185151 26036 rpcz_store.cc:276] 0807 11:11:07.434447 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:08.282876 (+848429us) service_pool.cc:227] Handling call
0807 11:11:08.606630 (+323754us) spinlock_profiling.cc:243] Waited 2.33 ms on lock 0x7b5800270998. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc461a2b0 00007f3fc46057f9 00007f3fc46052a2 00007f3fbdceb684
0807 11:11:08.606961 (+ 331us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:08.607692 (+ 731us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:08.607718 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:08.608281 (+ 563us) log.cc:842] Serialized 26 byte log entry
0807 11:11:08.740412 (+132131us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:08.740466 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:08.740572 (+ 106us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:11.171122 (+2430550us) spinlock_profiling.cc:243] Waited 61.7 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:11:11.184348 (+ 13226us) raft_consensus.cc:1671] finished
0807 11:11:11.184378 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:11.184537 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5305,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":172,"spinlock_wait_cycles":147329280,"thread_start_us":131671,"threads_started":1,"wal-append.queue_time_us":834897}
W20260807 11:11:11.877775 25931 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.722s user 0.005s sys 0.009s
I20260807 11:11:11.950659 25921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 371) took 1183 ms. Trace:
I20260807 11:11:10.294181 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:11:11.282186 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:11:11.573782 26985 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 497) took 2231 ms (client timeout 1680 ms). Trace:
I20260807 11:11:11.277218 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:11.598963 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:11.289938 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:11.966609 25921 rpcz_store.cc:276] 0807 11:11:10.767633 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.946320 (+178687us) service_pool.cc:227] Handling call
0807 11:11:11.082089 (+135769us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:11.082290 (+ 201us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:11.082317 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:11.082364 (+ 47us) raft_consensus.cc:1631] Marking committed up to 8
0807 11:11:11.082403 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:11.082469 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:11.950482 (+868013us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b540019ffd0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:11.950598 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":311517952}
I20260807 11:11:11.576741 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:11.971745 26034 raft_consensus.cc:2379] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 6: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
W20260807 11:11:11.949844 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 681416us
I20260807 11:11:11.974215 25869 leader_election.cc:302] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:11.275067 26033 raft_consensus.cc:2454] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
W20260807 11:11:11.950317 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 799790us
I20260807 11:11:11.336097 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:11.975359 25750 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:11.284114 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:11.976513 26042 raft_consensus.cc:2454] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 5.
I20260807 11:11:11.979000 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:11.979344 25655 catalog_manager.cc:5495] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 2 to 3, leader changed from eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 3 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260807 11:11:11.917263 30608 log.cc:924] Time spent T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.394s user 0.001s sys 0.000s
W20260807 11:11:11.918192 30609 log.cc:924] Time spent T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
W20260807 11:11:11.909336 30610 log.cc:924] Time spent T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.619s user 0.002s sys 0.000s
I20260807 11:11:11.982328 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:11.966814 26038 raft_consensus.cc:1261] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 11:11:11.221242 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:11.983253 25655 catalog_manager.cc:5495] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260807 11:11:11.221033 30582 consensus_queue.cc:1046] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.766s
W20260807 11:11:11.691943 30153 consensus_meta.cc:287] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 1.182s user 0.014s sys 0.002s
I20260807 11:11:12.005630 30153 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:12.008472 30153 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:11:11.852192 30391 consensus_meta.cc:287] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 1.194s user 0.004s sys 0.013s
I20260807 11:11:12.009511 25803 raft_consensus.cc:2454] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 8.
I20260807 11:11:11.341924 29851 raft_consensus.cc:482] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:11.988492 25926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1818) took 1821 ms. Trace:
I20260807 11:11:12.010028 30154 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:11:12.011166 30554 log.cc:924] Time spent T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 1.241s user 0.003s sys 0.015s
I20260807 11:11:12.010375 25926 rpcz_store.cc:276] 0807 11:11:10.167310 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.359599 (+192289us) service_pool.cc:227] Handling call
0807 11:11:10.429637 (+ 70038us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:11:10.677497 (+247860us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:10.677525 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 11:11:10.719063 (+ 41538us) log.cc:842] Serialized 52 byte log entry
0807 11:11:10.888509 (+169446us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:11:10.888559 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:10.888654 (+ 95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:11.988237 (+1099583us) raft_consensus.cc:1671] finished
0807 11:11:11.988258 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:11.988429 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":75291,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":14800,"thread_start_us":312419,"threads_started":2,"wal-append.queue_time_us":271027}
I20260807 11:11:12.010532 29851 raft_consensus.cc:504] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:11.290138 30547 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:11:11.810182 30475 consensus_meta.cc:287] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 1.327s user 0.010s sys 0.004s
W20260807 11:11:11.878100 30557 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.710s user 0.003s sys 0.012s
I20260807 11:11:11.361765 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
W20260807 11:11:11.989500 30351 consensus_meta.cc:287] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 1.021s user 0.011s sys 0.002s
I20260807 11:11:11.990906 26033 raft_consensus.cc:1226] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 5. Current term is 6. Ops: []
I20260807 11:11:11.991701 25919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 370) took 1230 ms. Trace:
I20260807 11:11:12.018893 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:11.376108 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:11.277402 26938 leader_election.cc:302] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:11.283246 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:11:11.965963 25931 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.810s user 0.007s sys 0.011s
W20260807 11:11:11.669557 26936 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.001s sys 0.014s
I20260807 11:11:11.278779 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:11.283664 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:11:11.968529 26985 rpcz_store.cc:269] 0807 11:11:09.342608 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:09.348407 (+ 5799us) service_pool.cc:227] Handling call
0807 11:11:11.505384 (+2156977us) spinlock_profiling.cc:243] Waited 930 ms on lock 0x7b5400741b54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:11.573693 (+ 68309us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2138589568}
I20260807 11:11:11.281497 30533 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 10 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:11.904997 30421 leader_election.cc:288] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:11:11.923537 30112 consensus_meta.cc:287] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 1.097s user 0.007s sys 0.003s
I20260807 11:11:11.995409 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1938) took 3059 ms. Trace:
I20260807 11:11:12.035434 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:12.065997 26038 rpcz_store.cc:276] 0807 11:11:08.936366 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.638905 (+1702539us) service_pool.cc:227] Handling call
0807 11:11:10.668269 (+ 29364us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:11.995054 (+1326785us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:11.995353 (+ 299us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:11.991784 25800 raft_consensus.cc:2454] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 6.
I20260807 11:11:12.033458 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:11.996021 30397 raft_consensus.cc:482] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:12.023995 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:11.993752 30544 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
W20260807 11:11:11.696889 30404 consensus_meta.cc:287] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 2.257s user 0.018s sys 0.000s
I20260807 11:11:12.010587 30154 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:12.085180 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:12.086033 25800 raft_consensus.cc:2396] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260807 11:11:11.277138 30415 raft_consensus.cc:2726] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:11.284327 30171 raft_consensus.cc:2781] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 11:11:12.016624 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59312 (request call id 1870) took 1268 ms. Trace:
I20260807 11:11:12.017120 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:11:12.019567 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 45429.4us
W20260807 11:11:12.064198 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 786830us
I20260807 11:11:12.090016 26039 rpcz_store.cc:276] 0807 11:11:10.748730 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:11.208622 (+459892us) service_pool.cc:227] Handling call
0807 11:11:12.016571 (+807949us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:12.090445 26034 raft_consensus.cc:2396] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 for term 9 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260807 11:11:12.066305 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1943) took 2988 ms. Trace:
I20260807 11:11:12.028294 25931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1830) took 1136 ms. Trace:
I20260807 11:11:12.057785 29991 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:12.044888 26036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 531) took 2509 ms. Trace:
I20260807 11:11:12.025789 30398 raft_consensus.cc:3032] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 LEADER]: Stepping down as leader of term 9
I20260807 11:11:12.063977 25799 raft_consensus.cc:2454] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 9.
I20260807 11:11:12.096307 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43246 (request call id 535) took 1229 ms. Trace:
I20260807 11:11:12.027673 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59312 (request call id 1872) took 1267 ms. Trace:
I20260807 11:11:12.096524 26036 rpcz_store.cc:276] 0807 11:11:09.535840 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:11.996373 (+2460533us) service_pool.cc:227] Handling call
0807 11:11:12.044104 (+ 47731us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:12.044379 (+ 275us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:12.044421 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:12.044480 (+ 59us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:12.044537 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:12.044610 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:12.044806 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:12.077104 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:12.102788 26038 raft_consensus.cc:3037] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:12.102872 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:11:12.018038 30557 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.850s user 0.003s sys 0.014s
I20260807 11:11:12.025219 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:12.079497 30397 raft_consensus.cc:504] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:12.064951 30391 raft_consensus.cc:2726] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 11:11:12.020538 30256 raft_consensus.cc:2726] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 11:11:12.018095 30324 raft_consensus.cc:3032] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:11:12.058024 25750 leader_election.cc:302] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:12.091054 26033 rpcz_store.cc:276] 0807 11:11:09.078196 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:11.990312 (+2912116us) service_pool.cc:227] Handling call
0807 11:11:11.990678 (+ 366us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:12.066015 (+ 75337us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:12.066251 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:11:11.716207 28964 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.009s sys 0.028s
I20260807 11:11:12.096901 26034 rpcz_store.cc:276] 0807 11:11:10.867082 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:11.973063 (+1105981us) service_pool.cc:227] Handling call
0807 11:11:12.096234 (+123171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:12.091226 25931 rpcz_store.cc:276] 0807 11:11:10.891600 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:11.026000 (+134400us) service_pool.cc:227] Handling call
0807 11:11:11.052401 (+ 26401us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:11:11.107342 (+ 54941us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:11.107394 (+ 52us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 11:11:11.112504 (+ 5110us) log.cc:842] Serialized 52 byte log entry
0807 11:11:12.027793 (+915289us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:11:12.027855 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:12.027954 (+ 99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:12.028022 (+ 68us) raft_consensus.cc:1671] finished
0807 11:11:12.028042 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:12.028208 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23566,"raft.queue_time_us":22671,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":17800,"thread_start_us":864594,"threads_started":1,"wal-append.queue_time_us":856027}
W20260807 11:11:12.072935 26936 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.021s user 0.004s sys 0.025s
W20260807 11:11:12.112707 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 1.08914e+06us
I20260807 11:11:12.098538 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:11:12.113409 28964 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.931s user 0.012s sys 0.044s
I20260807 11:11:12.113828 25797 raft_consensus.cc:2379] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 7: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
W20260807 11:11:12.114285 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 56404us
I20260807 11:11:12.099747 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:12.100267 25803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1869) took 1754 ms. Trace:
I20260807 11:11:12.095065 30171 raft_consensus.cc:482] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:12.101877 25918 raft_consensus.cc:2396] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260807 11:11:12.101748 25796 raft_consensus.cc:1261] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260807 11:11:12.018071 30475 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:12.103617 25800 raft_consensus.cc:2454] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 7.
W20260807 11:11:12.108994 30612 log.cc:924] Time spent T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.118s user 0.000s sys 0.001s
I20260807 11:11:12.058429 29966 consensus_queue.cc:1035] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.033s
I20260807 11:11:12.025017 30351 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:12.108294 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:12.097375 26037 rpcz_store.cc:276] 0807 11:11:10.759826 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:11.191941 (+432115us) service_pool.cc:227] Handling call
0807 11:11:12.027406 (+835465us) spinlock_profiling.cc:243] Waited 50.2 ms on lock 0x7b5400002b54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:12.027598 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":115447040}
I20260807 11:11:12.117750 30278 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:12.126426 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:12.127249 26037 raft_consensus.cc:2454] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:11:12.115621 25803 rpcz_store.cc:276] 0807 11:11:10.345569 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.376455 (+ 30886us) service_pool.cc:227] Handling call
0807 11:11:12.100071 (+1723616us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b5400265a54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:12.100191 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3624496384}
I20260807 11:11:12.073506 30400 raft_consensus.cc:2726] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:12.097959 30398 raft_consensus.cc:729] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:12.141494 25800 raft_consensus.cc:1226] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:12.120645 28964 raft_consensus.cc:2726] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 11:11:12.098073 30533 raft_consensus.cc:482] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:12.078480 30112 raft_consensus.cc:504] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:12.100077 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:12.131634 26034 raft_consensus.cc:1261] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 11:11:12.110296 30324 raft_consensus.cc:729] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:11:12.121883 30171 raft_consensus.cc:3037] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:12.147418 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:12.148231 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:12.099792 29991 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:12.095242 30544 raft_consensus.cc:504] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:12.019603 26046 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 3.
I20260807 11:11:12.173192 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59312 (request call id 1875) took 1147 ms. Trace:
I20260807 11:11:12.173759 26046 rpcz_store.cc:276] 0807 11:11:11.025739 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:11.995689 (+969950us) service_pool.cc:227] Handling call
0807 11:11:12.173105 (+177416us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:12.151285 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:12.118271 30404 consensus_queue.cc:1046] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20260807 11:11:12.038672 30547 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:12.153373 26035 raft_consensus.cc:2379] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 7: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:11:12.123842 30270 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:12.134104 30278 raft_consensus.cc:3037] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:12.157914 25869 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:12.183822 25801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1861) took 3114 ms. Trace:
I20260807 11:11:12.159925 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1865) took 1834 ms. Trace:
I20260807 11:11:12.161964 26980 raft_consensus.cc:2454] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:11:12.184276 25801 rpcz_store.cc:276] 0807 11:11:09.069549 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.504407 (+1434858us) service_pool.cc:227] Handling call
0807 11:11:11.216461 (+712054us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:11.217155 (+ 694us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:12.183624 (+966469us) spinlock_profiling.cc:243] Waited 693 ms on lock 0x7b54000e4cd0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:12.183761 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1594235648}
I20260807 11:11:12.161580 25798 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:11:12.120945 30390 consensus_queue.cc:1035] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.037s
I20260807 11:11:12.184661 26034 rpcz_store.cc:276] 0807 11:11:10.325814 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:12.111164 (+1785350us) service_pool.cc:227] Handling call
0807 11:11:12.131288 (+ 20124us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:12.159631 (+ 28343us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:12.159850 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:12.183490 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:12.186779 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:11:12.202159 30614 log.cc:924] Time spent T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.085s user 0.000s sys 0.001s
I20260807 11:11:12.207717 26984 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:11:12.207245 30324 consensus_queue.cc:260] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:12.186721 30398 consensus_queue.cc:260] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:12.222030 30324 raft_consensus.cc:3037] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:11:12.209879 30351 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:12.223676 30351 raft_consensus.cc:482] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:12.224164 30351 raft_consensus.cc:504] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:12.229187 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:12.230068 26046 raft_consensus.cc:2454] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:11:12.176194 29851 leader_election.cc:288] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:11:12.234045 25750 connection.cc:232] RPC call timeout handler was delayed by -1.3891391139477491s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1532s for a response
W20260807 11:11:12.238041 25750 connection.cc:232] RPC call timeout handler was delayed by -1.3495391139294952s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15760000000000002s for a response
W20260807 11:11:12.238608 25750 connection.cc:232] RPC call timeout handler was delayed by -1.2379391139838845s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17s for a response
W20260807 11:11:12.239030 25750 connection.cc:232] RPC call timeout handler was delayed by -1.2109391139820218s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17300000000000001s for a response
I20260807 11:11:12.241526 26038 raft_consensus.cc:2454] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:11:12.232230 30557 raft_consensus.cc:2726] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:12.211243 30112 leader_election.cc:288] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:12.257215 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:12.162715 30533 raft_consensus.cc:504] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:12.258563 25750 leader_election.cc:302] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:12.259469 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:12.377019 26046 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:11:12.210659 29991 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:11:12.367424 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 209576us
I20260807 11:11:12.252215 29857 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:12.372375 29851 consensus_queue.cc:1035] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 11:11:12.209040 30270 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:12.231724 30154 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:12.251796 28964 raft_consensus.cc:482] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:12.419605 26034 raft_consensus.cc:3037] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:12.426381 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:12.444561 25869 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:11:12.449182 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 190714us
I20260807 11:11:12.450263 25750 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:12.451864 25750 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:12.417059 30562 raft_consensus.cc:2726] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:12.470291 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 537) took 1592 ms. Trace:
I20260807 11:11:12.473449 26046 rpcz_store.cc:276] 0807 11:11:10.877875 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:12.448069 (+1570194us) service_pool.cc:227] Handling call
0807 11:11:12.469631 (+ 21562us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:12.469888 (+ 257us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:12.469928 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:12.469989 (+ 61us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:12.470036 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:12.470085 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:12.470219 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:12.413905 29857 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:12.479943 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 533) took 2214 ms. Trace:
I20260807 11:11:12.471285 30404 raft_consensus.cc:2726] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:12.468971 30475 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:12.545575 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1874) took 1456 ms. Trace:
I20260807 11:11:12.549958 26046 rpcz_store.cc:276] 0807 11:11:11.022469 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:12.475111 (+1452642us) service_pool.cc:227] Handling call
0807 11:11:12.478724 (+ 3613us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:12.478891 (+ 167us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:12.478911 (+ 20us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:12.478947 (+ 36us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:11:12.478979 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:12.479024 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:12.479148 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:12.547820 26039 rpcz_store.cc:276] 0807 11:11:10.265407 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:12.090848 (+1825441us) service_pool.cc:227] Handling call
0807 11:11:12.104740 (+ 13892us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:12.479541 (+374801us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:12.479581 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:12.479628 (+ 47us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:12.479681 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:12.479759 (+ 78us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:12.479880 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":271776,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":28229,"thread_start_us":361653,"threads_started":1}
W20260807 11:11:12.559614 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 436714us
W20260807 11:11:12.568697 25981 connection.cc:232] RPC call timeout handler was delayed by -2.5423085042275488s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1595s for a response
W20260807 11:11:12.569167 25981 connection.cc:232] RPC call timeout handler was delayed by -2.5144085041247308s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1626s for a response
W20260807 11:11:12.572080 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 120325us
W20260807 11:11:12.597621 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 173787us
I20260807 11:11:12.599728 25869 leader_election.cc:302] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:11:12.569547 25981 connection.cc:232] RPC call timeout handler was delayed by -2.5099085040856153s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16310000000000002s for a response
W20260807 11:11:12.601348 25981 connection.cc:232] RPC call timeout handler was delayed by -2.4280085042119026s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17220000000000002s for a response
W20260807 11:11:12.601748 25981 connection.cc:232] RPC call timeout handler was delayed by -2.421708504203707s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17290000000000003s for a response
W20260807 11:11:12.602084 25981 connection.cc:232] RPC call timeout handler was delayed by -2.2642085042316467s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1904s for a response
W20260807 11:11:12.602411 25981 connection.cc:232] RPC call timeout handler was delayed by -2.2408085041679442s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.193s for a response
W20260807 11:11:12.602743 25981 connection.cc:232] RPC call timeout handler was delayed by -2.237208504229784s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19340000000000002s for a response
W20260807 11:11:12.603072 25981 connection.cc:232] RPC call timeout handler was delayed by -2.1904085041023791s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1986s for a response
W20260807 11:11:12.603313 25981 connection.cc:232] RPC call timeout handler was delayed by -2.1643085042014718s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2015s for a response
W20260807 11:11:12.603581 25981 connection.cc:232] RPC call timeout handler was delayed by -2.1643085042014718s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2015s for a response
W20260807 11:11:12.603931 25981 connection.cc:232] RPC call timeout handler was delayed by -2.1184085041750222s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2066s for a response
W20260807 11:11:12.604247 25981 connection.cc:232] RPC call timeout handler was delayed by -2.1085085042286664s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2077s for a response
W20260807 11:11:12.606953 25981 connection.cc:232] RPC call timeout handler was delayed by -2.0842085040640086s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21040000000000003s for a response
W20260807 11:11:12.607385 25981 connection.cc:232] RPC call timeout handler was delayed by -2.0842085040640086s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21040000000000003s for a response
W20260807 11:11:12.607827 25981 connection.cc:232] RPC call timeout handler was delayed by -2.059008504031226s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21320000000000003s for a response
W20260807 11:11:12.608309 25981 connection.cc:232] RPC call timeout handler was delayed by -2.0356085042003542s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2158s for a response
W20260807 11:11:12.608883 25981 connection.cc:232] RPC call timeout handler was delayed by -2.0140085041057318s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2182s for a response
W20260807 11:11:12.610000 25981 connection.cc:232] RPC call timeout handler was delayed by -2.0140085041057318s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2182s for a response
W20260807 11:11:12.610675 25981 connection.cc:232] RPC call timeout handler was delayed by -1.8268085040617734s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23900000000000002s for a response
W20260807 11:11:12.611380 25981 connection.cc:232] RPC call timeout handler was delayed by -1.8268085040617734s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23900000000000002s for a response
W20260807 11:11:12.603320 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 456111us
I20260807 11:11:12.531567 29915 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:12.614624 29915 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:11:12.614006 25981 connection.cc:232] RPC call timeout handler was delayed by -1.7980085040908307s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24220000000000003s for a response
W20260807 11:11:12.619894 25981 connection.cc:232] RPC call timeout handler was delayed by -1.6837085040751845s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2549s for a response
W20260807 11:11:12.620405 25981 connection.cc:232] RPC call timeout handler was delayed by -1.5487085040658712s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2699s for a response
W20260807 11:11:12.620846 25981 connection.cc:232] RPC call timeout handler was delayed by -1.4449471321422607s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19840000000000002s for a response
W20260807 11:11:12.621564 25981 connection.cc:232] RPC call timeout handler was delayed by -1.3531471320893615s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2086s for a response
W20260807 11:11:12.622076 25981 connection.cc:232] RPC call timeout handler was delayed by -1.3336085041519254s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2938s for a response
W20260807 11:11:12.622498 25981 connection.cc:232] RPC call timeout handler was delayed by -1.3183085040654987s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.29550000000000004s for a response
W20260807 11:11:12.622907 26042 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.025s sys 0.085s
W20260807 11:11:12.623610 26042 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.528s user 0.025s sys 0.085s
W20260807 11:11:12.623245 25981 connection.cc:232] RPC call timeout handler was delayed by -1.2964471322484314s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2149s for a response
W20260807 11:11:12.624725 25981 connection.cc:232] RPC call timeout handler was delayed by -1.2748471321538091s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21730000000000002s for a response
W20260807 11:11:12.625306 25981 connection.cc:232] RPC call timeout handler was delayed by -1.0627085040323436s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3239s for a response
I20260807 11:11:12.636085 30421 raft_consensus.cc:2726] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:11:12.638115 30404 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:12.472429 28964 raft_consensus.cc:504] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:12.613411 30475 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:12.612751 29991 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:12.613610 30112 raft_consensus.cc:482] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:12.625756 29966 raft_consensus.cc:2726] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 11:11:12.630798 30320 leader_election.cc:288] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:12.638934 30397 leader_election.cc:288] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:12.638640 30404 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:12.618420 30544 leader_election.cc:288] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:12.632539 30351 leader_election.cc:288] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:12.727918 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1855) took 4086 ms. Trace:
I20260807 11:11:12.742676 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1866) took 2405 ms. Trace:
W20260807 11:11:12.733049 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 158819us
I20260807 11:11:12.740645 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:12.719797 25803 raft_consensus.cc:1226] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260807 11:11:12.716833 25981 leader_election.cc:302] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:12.802776 25750 leader_election.cc:302] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:12.799418 26041 rpcz_store.cc:276] 0807 11:11:08.643442 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.370491 (+1727049us) service_pool.cc:227] Handling call
0807 11:11:11.145849 (+775358us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:11.146442 (+ 593us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:12.727693 (+1581251us) spinlock_profiling.cc:243] Waited 725 ms on lock 0x7b54000bb650. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:12.727851 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1668618624}
I20260807 11:11:12.710752 25869 leader_election.cc:302] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:12.717443 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:12.739037 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:12.700644 26034 raft_consensus.cc:2396] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260807 11:11:12.800271 25799 raft_consensus.cc:2454] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
W20260807 11:11:12.789482 30585 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.012s sys 0.029s
I20260807 11:11:12.799664 26037 rpcz_store.cc:276] 0807 11:11:10.337064 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:12.127960 (+1790896us) service_pool.cc:227] Handling call
0807 11:11:12.144204 (+ 16244us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:12.144496 (+ 292us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 11:11:12.461556 (+317060us) log.cc:842] Serialized 14 byte log entry
0807 11:11:12.742231 (+280675us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:12.742309 (+ 78us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:12.742379 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:12.742463 (+ 84us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:12.742607 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":594408,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":82,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":129,"thread_start_us":589059,"threads_started":2}
W20260807 11:11:12.749506 30278 consensus_meta.cc:287] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.541s user 0.008s sys 0.003s
I20260807 11:11:12.806921 25801 raft_consensus.cc:2454] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 9.
I20260807 11:11:12.691429 26974 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50942 (request call id 498) took 1489 ms. Trace:
I20260807 11:11:12.820294 25750 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:12.820914 30415 raft_consensus.cc:2726] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260807 11:11:12.809679 26987 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.593s user 0.008s sys 0.005s
I20260807 11:11:12.725983 29915 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:12.732506 30533 leader_election.cc:288] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:11:12.833417 26987 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s user 0.008s sys 0.005s
I20260807 11:11:12.832971 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:12.832971 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:12.837777 26037 raft_consensus.cc:2454] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
W20260807 11:11:12.832952 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 122331us
W20260807 11:11:12.810024 30552 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.607s user 0.009s sys 0.003s
I20260807 11:11:12.834455 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:11:12.817335 30556 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.645s user 0.003s sys 0.009s
I20260807 11:11:12.750831 30648 consensus_queue.cc:1046] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.080s
I20260807 11:11:12.832300 26974 rpcz_store.cc:276] 0807 11:11:11.197446 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:11.198407 (+ 961us) service_pool.cc:227] Handling call
0807 11:11:11.198806 (+ 399us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:11.210348 (+ 11542us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 11:11:11.210692 (+ 344us) log.cc:842] Serialized 14 byte log entry
0807 11:11:11.985344 (+774652us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:11.985418 (+ 74us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:11.985479 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:12.630730 (+645251us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b540068c254. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:12.632954 (+ 2224us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:12.686979 (+ 54025us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":297083,"prepare.queue_time_us":241,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":298284,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":100,"spinlock_wait_cycles":25965824,"thread_start_us":456547,"threads_started":1,"wal-append.queue_time_us":790493}
W20260807 11:11:12.817484 30585 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.563s user 0.012s sys 0.029s
I20260807 11:11:12.839402 25869 leader_election.cc:302] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
W20260807 11:11:12.919042 25786 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.706s user 0.010s sys 0.021s
I20260807 11:11:12.837935 26041 raft_consensus.cc:3037] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
W20260807 11:11:12.921358 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 82059.5us
W20260807 11:11:12.919274 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 352764us
W20260807 11:11:12.921370 25786 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.708s user 0.010s sys 0.021s
I20260807 11:11:12.693892 30112 raft_consensus.cc:504] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:12.690320 30475 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:12.700862 29991 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:12.918341 30556 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.746s user 0.009s sys 0.009s
I20260807 11:11:12.924544 25869 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:12.836632 30320 raft_consensus.cc:2726] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 11:11:12.953985 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:12.836270 30397 raft_consensus.cc:2726] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:12.955340 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 532) took 2690 ms. Trace:
W20260807 11:11:12.918006 30552 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.715s user 0.009s sys 0.004s
I20260807 11:11:12.959751 26042 rpcz_store.cc:276] 0807 11:11:10.264996 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:11.992160 (+1727164us) service_pool.cc:227] Handling call
0807 11:11:12.037922 (+ 45762us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:12.076951 (+ 39029us) raft_consensus.cc:1520] Early marking committed up to index 6
0807 11:11:12.625693 (+548742us) log.cc:842] Serialized 14 byte log entry
0807 11:11:12.954988 (+329295us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:12.955039 (+ 51us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:12.955087 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:12.955149 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:12.955285 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1478,"prepare.queue_time_us":616955,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":44,"raft.queue_time_us":25675,"thread_start_us":742343,"threads_started":2}
I20260807 11:11:12.825819 30278 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:11:12.961099 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 140895us
I20260807 11:11:12.966459 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:12.967674 26037 raft_consensus.cc:2396] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 11:11:12.951118 30654 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:12.955413 30415 raft_consensus.cc:2726] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:12.978196 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 540) took 1812 ms. Trace:
I20260807 11:11:12.978639 26042 rpcz_store.cc:276] 0807 11:11:11.161744 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:12.963632 (+1801888us) service_pool.cc:227] Handling call
0807 11:11:12.973451 (+ 9819us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:12.973687 (+ 236us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:12.973712 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:12.973763 (+ 51us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:12.973813 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:12.973906 (+ 93us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:12.974039 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:11:12.978682 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 300589us
W20260807 11:11:12.983546 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 368960us
I20260807 11:11:12.969170 30320 raft_consensus.cc:2781] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260807 11:11:12.982493 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 349092us
I20260807 11:11:13.001715 30585 raft_consensus.cc:3032] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 11:11:13.002295 30585 raft_consensus.cc:729] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:13.003834 30585 consensus_queue.cc:260] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:13.007514 30585 raft_consensus.cc:3037] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:12.968986 29857 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:13.011462 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1948) took 1681 ms. Trace:
I20260807 11:11:13.012221 26037 rpcz_store.cc:276] 0807 11:11:11.330450 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:12.977408 (+1646958us) service_pool.cc:227] Handling call
0807 11:11:13.010824 (+ 33416us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:13.011047 (+ 223us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:13.011083 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:13.011138 (+ 55us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:13.011186 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:13.011252 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:13.011390 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:12.925071 30154 raft_consensus.cc:2726] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:12.927553 30544 raft_consensus.cc:2781] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260807 11:11:13.020391 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 54118.5us
I20260807 11:11:12.973385 30496 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
W20260807 11:11:13.043318 30664 log.cc:924] Time spent T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20260807 11:11:13.043372 30256 raft_consensus.cc:3032] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 7 LEADER]: Stepping down as leader of term 7
I20260807 11:11:13.043963 30256 raft_consensus.cc:729] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:12.981215 30654 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:13.044850 30256 consensus_queue.cc:260] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:13.046124 30256 raft_consensus.cc:3037] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:13.047585 30654 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:13.048175 30654 consensus_queue.cc:1035] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260807 11:11:12.993016 30320 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:13.049327 30320 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Advancing to term 10
I20260807 11:11:13.055956 30112 leader_election.cc:288] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:11:13.056984 30656 log.cc:924] Time spent T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.140s user 0.001s sys 0.000s
I20260807 11:11:13.057612 25750 leader_election.cc:302] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:13.058425 25797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1883) took 1116 ms. Trace:
I20260807 11:11:13.058781 29857 raft_consensus.cc:2781] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 11:11:13.059458 29857 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:13.059929 29857 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Advancing to term 9
I20260807 11:11:13.060009 30404 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:13.060915 29991 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:13.058902 25786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1882) took 1776 ms. Trace:
I20260807 11:11:13.075268 26041 raft_consensus.cc:2454] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 7.
I20260807 11:11:13.065088 25786 rpcz_store.cc:276] 0807 11:11:11.282513 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:12.113125 (+830612us) service_pool.cc:227] Handling call
0807 11:11:12.130781 (+ 17656us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:11:12.144830 (+ 14049us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:12.144863 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 11:11:12.177585 (+ 32722us) log.cc:842] Serialized 52 byte log entry
0807 11:11:12.924564 (+746979us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:11:12.924612 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:12.924680 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:13.058700 (+134020us) raft_consensus.cc:1671] finished
0807 11:11:13.058720 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:13.058847 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2069,"raft.queue_time_us":7008,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":17615,"thread_start_us":711106,"threads_started":1,"wal-append.queue_time_us":774255}
I20260807 11:11:13.080636 26037 raft_consensus.cc:1226] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 7. Current term is 8. Ops: []
I20260807 11:11:13.070418 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 539) took 1988 ms. Trace:
I20260807 11:11:13.081663 26039 rpcz_store.cc:276] 0807 11:11:11.081675 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:12.552479 (+1470804us) service_pool.cc:227] Handling call
0807 11:11:12.626638 (+ 74159us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:12.644567 (+ 17929us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:12.644611 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:12.695188 (+ 50577us) log.cc:842] Serialized 26 byte log entry
0807 11:11:12.976469 (+281281us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:12.976519 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:12.976601 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:13.070153 (+ 93552us) raft_consensus.cc:1671] finished
0807 11:11:13.070179 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:13.070348 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":92087,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":17730,"thread_start_us":242175,"threads_started":1,"wal-append.queue_time_us":339300}
I20260807 11:11:13.078715 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1867) took 2331 ms. Trace:
I20260807 11:11:13.087844 26038 rpcz_store.cc:276] 0807 11:11:10.747767 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:12.242453 (+1494686us) service_pool.cc:227] Handling call
0807 11:11:12.242849 (+ 396us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:12.243114 (+ 265us) raft_consensus.cc:1520] Early marking committed up to index 5
0807 11:11:12.828254 (+585140us) log.cc:842] Serialized 14 byte log entry
0807 11:11:13.078268 (+250014us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:13.078351 (+ 83us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:11:13.078425 (+ 74us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:13.078506 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:13.078648 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":503089,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":602,"thread_start_us":710518,"threads_started":2}
I20260807 11:11:13.091470 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1950) took 1094 ms. Trace:
I20260807 11:11:13.092113 26037 rpcz_store.cc:276] 0807 11:11:11.996835 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:13.026865 (+1030030us) service_pool.cc:227] Handling call
0807 11:11:13.080411 (+ 53546us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:13.091122 (+ 10711us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:13.091389 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:13.058830 25797 rpcz_store.cc:276] 0807 11:11:11.941744 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:12.114422 (+172678us) service_pool.cc:227] Handling call
0807 11:11:12.114761 (+ 339us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:12.144225 (+ 29464us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:12.144263 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:12.157956 (+ 13693us) log.cc:842] Serialized 26 byte log entry
0807 11:11:12.618536 (+460580us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:12.618579 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:12.618656 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:13.058193 (+439537us) raft_consensus.cc:1671] finished
0807 11:11:13.058212 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:13.058361 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6750,"raft.queue_time_us":24021,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":10606,"thread_start_us":413218,"threads_started":1,"wal-append.queue_time_us":568715}
I20260807 11:11:13.010191 28964 leader_election.cc:288] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:13.036020 30544 raft_consensus.cc:482] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 11:11:13.117614 25872 leader_election.cc:334] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009): Timed out: RequestConsensusVote RPC to 127.23.120.131:46009 timed out after 1.696s (SENT)
I20260807 11:11:13.120955 25872 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:13.136505 30557 raft_consensus.cc:2726] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 11:11:13.120998 30544 raft_consensus.cc:3037] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:11:13.039209 30496 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:13.142815 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 184730us
I20260807 11:11:13.141004 29847 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:13.146996 26938 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:13.105018 30415 consensus_queue.cc:1046] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.038s
I20260807 11:11:13.152196 30675 raft_consensus.cc:2726] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 11:11:13.159004 30270 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:11:13.162194 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 237691us
I20260807 11:11:13.177060 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:11:13.177295 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 155458us
I20260807 11:11:13.178563 26041 raft_consensus.cc:2454] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:11:13.182261 25981 leader_election.cc:302] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:13.185015 26987 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 512) took 1032 ms. Trace:
I20260807 11:11:13.191072 25869 leader_election.cc:302] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:13.192492 25869 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:11:13.185407 26987 rpcz_store.cc:276] 0807 11:11:12.152491 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:12.152655 (+ 164us) service_pool.cc:227] Handling call
0807 11:11:12.172640 (+ 19985us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:12.184153 (+ 11513us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:12.184190 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:12.216008 (+ 31818us) log.cc:842] Serialized 26 byte log entry
0807 11:11:12.837694 (+621686us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:12.944306 (+106612us) log.cc:842] Serialized 14 byte log entry
0807 11:11:13.100192 (+155886us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:13.100283 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:13.184801 (+ 84518us) raft_consensus.cc:1671] finished
0807 11:11:13.184823 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:13.184959 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13047,"prepare.queue_time_us":33738,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":119557,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":169,"thread_start_us":765330,"threads_started":2,"wal-append.queue_time_us":848747}
I20260807 11:11:13.195564 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:13.197721 26041 raft_consensus.cc:2379] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 7: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
W20260807 11:11:13.195523 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 211313us
I20260807 11:11:13.202517 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:13.204171 25869 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:13.148011 29847 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:13.193988 30475 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:13.195482 30278 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:13.204020 29991 raft_consensus.cc:2726] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 11:11:13.225060 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1864) took 2899 ms. Trace:
I20260807 11:11:12.018976 25919 rpcz_store.cc:276] 0807 11:11:10.761622 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.911874 (+150252us) service_pool.cc:227] Handling call
0807 11:11:10.912258 (+ 384us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:11.337280 (+425022us) raft_consensus.cc:1520] Early marking committed up to index 7
0807 11:11:11.574306 (+237026us) log.cc:842] Serialized 14 byte log entry
0807 11:11:11.991367 (+417061us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:11.991423 (+ 56us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:11.991472 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:11.991540 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:11.991646 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":274243,"prepare.queue_time_us":236768,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":60,"raft.queue_time_us":224842,"raft.run_cpu_time_us":294,"raft.run_wall_time_us":46390,"thread_start_us":182123,"threads_started":1,"wal-append.queue_time_us":388341}
I20260807 11:11:13.330979 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:13.218946 25799 raft_consensus.cc:1261] T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260807 11:11:13.237782 30496 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:13.710834 25921 raft_consensus.cc:2454] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:11:13.241904 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:13.712101 25931 raft_consensus.cc:2379] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 6: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
W20260807 11:11:13.234788 30679 log.cc:924] Time spent T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20260807 11:11:13.715212 30496 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:13.208156 30256 raft_consensus.cc:2781] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 11:11:13.715855 30496 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:13.716395 26036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1863) took 3399 ms. Trace:
W20260807 11:11:13.680891 30704 log.cc:924] Time spent T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
W20260807 11:11:13.357829 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 255946us
I20260807 11:11:13.716979 26036 rpcz_store.cc:276] 0807 11:11:10.318786 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:12.103315 (+1784529us) service_pool.cc:227] Handling call
0807 11:11:12.116266 (+ 12951us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:12.588964 (+472698us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:12.589020 (+ 56us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:12.612252 (+ 23232us) log.cc:842] Serialized 26 byte log entry
0807 11:11:12.962398 (+350146us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:12.962442 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:12.962508 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:13.716148 (+753640us) raft_consensus.cc:1671] finished
0807 11:11:13.716174 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:13.716326 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":268811,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":26026,"thread_start_us":694550,"threads_started":2,"wal-append.queue_time_us":490337}
I20260807 11:11:13.213760 30557 raft_consensus.cc:2781] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 11:11:13.713274 29851 consensus_queue.cc:1046] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:11:13.196008 30270 raft_consensus.cc:3037] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:13.713982 30696 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:13.720356 29851 raft_consensus.cc:3032] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 11:11:13.352301 25806 raft_consensus.cc:1261] T 9c6408e9acee46adacd2bbd81669f27a P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260807 11:11:13.714304 25752 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:13.720971 26046 raft_consensus.cc:1226] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:13.721483 30557 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:13.721872 30557 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Advancing to term 9
I20260807 11:11:13.722196 30693 raft_consensus.cc:482] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:13.722779 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1944) took 2647 ms. Trace:
I20260807 11:11:13.722990 30693 raft_consensus.cc:504] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:13.723254 26046 rpcz_store.cc:276] 0807 11:11:11.075408 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:12.551028 (+1475620us) service_pool.cc:227] Handling call
0807 11:11:13.720782 (+1169754us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:13.721520 (+ 738us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:13.722596 (+ 1076us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54000569d0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:13.722720 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2536986496}
I20260807 11:11:13.724750 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:13.725737 30693 leader_election.cc:288] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:13.726037 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:13.726855 26046 raft_consensus.cc:2454] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 8.
I20260807 11:11:13.718272 30496 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:13.727429 25796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 419) took 4953 ms. Trace:
I20260807 11:11:13.719116 29966 consensus_queue.cc:1035] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:11:13.720417 30696 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:13.730212 29966 raft_consensus.cc:2726] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:13.730876 29966 consensus_queue.cc:1035] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 11:11:13.730885 30696 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:13.216629 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:13.720938 29851 raft_consensus.cc:729] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:13.732259 26041 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:11:13.733258 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:13.732897 29851 consensus_queue.cc:260] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:13.335180 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 86670.6us
I20260807 11:11:13.733147 25796 rpcz_store.cc:276] 0807 11:11:08.774098 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.464551 (+1690453us) service_pool.cc:227] Handling call
0807 11:11:12.100736 (+1636185us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:12.119128 (+ 18392us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:13.727250 (+1608122us) spinlock_profiling.cc:243] Waited 1.62 s on lock 0x7b5400265a50. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:13.727373 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3725229440}
I20260807 11:11:13.735085 30717 consensus_queue.cc:1046] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 11:11:13.735563 30398 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 FOLLOWER]: Advancing to term 10
I20260807 11:11:13.716488 30256 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:13.327209 30320 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:13.736373 30256 raft_consensus.cc:3037] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Advancing to term 8
W20260807 11:11:13.236076 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 53906.5us
I20260807 11:11:13.737449 25981 leader_election.cc:398] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 election: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6.
I20260807 11:11:13.737818 30320 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 10 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:13.737962 25981 leader_election.cc:401] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260807 11:11:13.728121 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:13.739950 30557 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:13.739905 25981 leader_election.cc:398] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 7. Current term is 8.
I20260807 11:11:13.744583 25981 leader_election.cc:401] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260807 11:11:13.733948 26041 raft_consensus.cc:2454] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 7.
W20260807 11:11:13.711356 25921 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1653) took 1719 ms (client timeout 1929 ms). Trace:
I20260807 11:11:13.735474 25869 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:11:13.745671 25921 rpcz_store.cc:269] 0807 11:11:11.992042 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:13.326857 (+1334815us) service_pool.cc:227] Handling call
0807 11:11:13.711300 (+384443us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:13.437640 26978 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50958 (request call id 500) took 3109 ms. Trace:
I20260807 11:11:13.746726 30696 raft_consensus.cc:2781] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:11:13.746961 25869 leader_election.cc:302] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
W20260807 11:11:13.684583 30585 consensus_meta.cc:287] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.677s user 0.010s sys 0.001s
I20260807 11:11:13.747143 30696 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:13.334909 30415 raft_consensus.cc:482] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:13.747932 30696 raft_consensus.cc:3037] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:11:13.748135 30421 consensus_queue.cc:1046] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:11:13.748301 30415 raft_consensus.cc:504] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:13.745739 26046 raft_consensus.cc:2362] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6. Re-sending same reply.
I20260807 11:11:13.749112 30421 raft_consensus.cc:2781] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:13.749482 30421 raft_consensus.cc:482] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:13.749847 30421 raft_consensus.cc:3037] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:13.749794 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:13.750710 26046 raft_consensus.cc:2379] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 10: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:11:13.751492 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:13.751683 25869 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:11:13.752095 30415 leader_election.cc:288] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:11:13.745744 26035 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59312 (request call id 1876) took 2712 ms (client timeout 2300 ms). Trace:
I20260807 11:11:13.752748 30278 raft_consensus.cc:2781] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260807 11:11:13.687628 29857 consensus_meta.cc:287] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.627s user 0.009s sys 0.001s
I20260807 11:11:13.753476 30278 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:13.754299 30278 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Advancing to term 10
I20260807 11:11:13.754148 29857 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:13.746402 26978 rpcz_store.cc:276] 0807 11:11:10.328456 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.346786 (+ 18330us) service_pool.cc:227] Handling call
0807 11:11:13.437369 (+3090583us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54005ffad4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:13.437569 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2999369088}
I20260807 11:11:13.755321 25801 raft_consensus.cc:1226] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 8. Current term is 9. Ops: []
W20260807 11:11:13.753255 26035 rpcz_store.cc:269] 0807 11:11:11.033515 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:11.293338 (+259823us) service_pool.cc:227] Handling call
0807 11:11:13.720644 (+2427306us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b54000569d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:13.745668 (+ 25024us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":299391872}
W20260807 11:11:13.707986 26936 connection.cc:232] RPC call timeout handler was delayed by -2.1843407019041479s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2722s for a response
I20260807 11:11:13.757651 26035 raft_consensus.cc:1261] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260807 11:11:13.758827 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:13.759305 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:13.761698 26035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1958) took 1558 ms. Trace:
I20260807 11:11:13.762181 26035 rpcz_store.cc:276] 0807 11:11:12.203495 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:13.756806 (+1553311us) service_pool.cc:227] Handling call
0807 11:11:13.757182 (+ 376us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:13.761434 (+ 4252us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:13.761641 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:13.357018 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 1959) took 1133 ms. Trace:
I20260807 11:11:13.359568 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:13.781940 26034 rpcz_store.cc:276] 0807 11:11:12.203863 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:12.233035 (+ 29172us) service_pool.cc:227] Handling call
0807 11:11:13.337364 (+1104329us) spinlock_profiling.cc:243] Waited 48.7 ms on lock 0x7b540009f454. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:13.337592 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":111929728}
W20260807 11:11:13.707465 25918 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1822) took 3423 ms (client timeout 1730 ms). Trace:
I20260807 11:11:13.709717 26033 rpcz_store.cc:276] 0807 11:11:10.325467 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:12.108498 (+1783031us) service_pool.cc:227] Handling call
0807 11:11:12.115365 (+ 6867us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:12.547292 (+431927us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:12.547342 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:12.609852 (+ 62510us) log.cc:842] Serialized 26 byte log entry
0807 11:11:13.013475 (+403623us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:13.013515 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:13.013584 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:13.224838 (+211254us) raft_consensus.cc:1671] finished
0807 11:11:13.224869 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:13.225001 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":301542,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":20023,"thread_start_us":778490,"threads_started":2,"wal-append.queue_time_us":575296}
W20260807 11:11:13.784659 25918 rpcz_store.cc:269] 0807 11:11:10.288972 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:10.315636 (+ 26664us) service_pool.cc:227] Handling call
0807 11:11:13.707240 (+3391604us) spinlock_profiling.cc:243] Waited 1.75 s on lock 0x7b54003b1ad4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:13.707395 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4026065152}
W20260807 11:11:13.563086 30324 consensus_meta.cc:287] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 1.340s user 0.008s sys 0.009s
W20260807 11:11:13.807924 25808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59118 (request call id 423) took 4971 ms (client timeout 2722 ms). Trace:
W20260807 11:11:13.812045 25808 rpcz_store.cc:269] 0807 11:11:08.810508 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:09.349824 (+539316us) service_pool.cc:227] Handling call
0807 11:11:13.781246 (+4431422us) spinlock_profiling.cc:243] Waited 967 ms on lock 0x7b54002d46d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:13.781387 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2224713984}
W20260807 11:11:13.813577 26039 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.001s sys 0.016s
W20260807 11:11:13.366658 30685 log.cc:924] Time spent T a3d806783f4844d782ffd0c901a1dafb P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.113s user 0.000s sys 0.001s
W20260807 11:11:13.783516 30630 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.622s user 0.001s sys 0.008s
W20260807 11:11:13.829816 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 90047us
I20260807 11:11:13.767290 29991 raft_consensus.cc:2726] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 7. Current term is 8.
I20260807 11:11:13.816320 25800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1896) took 1178 ms. Trace:
W20260807 11:11:13.816804 26039 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.612s user 0.001s sys 0.017s
I20260807 11:11:13.816689 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:13.818163 25916 raft_consensus.cc:1261] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260807 11:11:13.816205 25920 raft_consensus.cc:1226] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:13.820950 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:13.824441 25926 raft_consensus.cc:1261] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260807 11:11:13.826005 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 72732.4us
I20260807 11:11:13.826821 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:13.815464 25917 raft_consensus.cc:1261] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 7. (term mismatch)
I20260807 11:11:13.844339 25917 pending_rounds.cc:76] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc: Aborting all ops after (but not including) 6
I20260807 11:11:13.844661 25917 pending_rounds.cc:97] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc: Aborting uncommitted NO_OP operation due to leader change: 7.7
I20260807 11:11:13.844990 25917 raft_consensus.cc:2866] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:11:13.756563 29857 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 9 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:13.833948 26041 raft_consensus.cc:1261] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260807 11:11:13.859304 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1963) took 1486 ms. Trace:
I20260807 11:11:13.859768 26041 rpcz_store.cc:276] 0807 11:11:12.378512 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:13.802706 (+1424194us) service_pool.cc:227] Handling call
0807 11:11:13.833612 (+ 30906us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:13.858967 (+ 25355us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:13.859234 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:13.834215 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:11:13.833900 26037 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.641s user 0.007s sys 0.048s
W20260807 11:11:13.861490 26037 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.679s user 0.007s sys 0.048s
I20260807 11:11:13.834762 25932 raft_consensus.cc:1226] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260807 11:11:13.853883 25918 raft_consensus.cc:1226] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 10 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 9. Current term is 10. Ops: []
W20260807 11:11:13.819707 30171 consensus_meta.cc:287] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 1.639s user 0.012s sys 0.003s
I20260807 11:11:13.855378 26038 raft_consensus.cc:1261] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 11:11:13.861055 25796 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:11:13.867272 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:13.868274 25806 raft_consensus.cc:2454] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
W20260807 11:11:13.861246 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 102538us
I20260807 11:11:13.333099 29915 consensus_queue.cc:1046] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.527s
W20260807 11:11:13.828918 30630 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.667s user 0.002s sys 0.008s
I20260807 11:11:13.873559 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:13.836987 29847 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:13.887744 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:11:13.888528 26042 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.823s user 0.028s sys 0.048s
I20260807 11:11:13.883538 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:11:13.888984 26042 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.823s user 0.028s sys 0.048s
I20260807 11:11:13.882947 26041 raft_consensus.cc:1261] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 10 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260807 11:11:13.869812 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:13.889559 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1966) took 1411 ms. Trace:
I20260807 11:11:13.889906 26041 rpcz_store.cc:276] 0807 11:11:12.478381 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:13.860546 (+1382165us) service_pool.cc:227] Handling call
0807 11:11:13.882575 (+ 22029us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:13.889334 (+ 6759us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:13.889507 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:13.890970 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:13.854588 29417 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:13.895562 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:13.899003 25981 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:11:13.900537 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 180654us
I20260807 11:11:13.897272 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1949) took 2425 ms. Trace:
I20260807 11:11:13.901610 26042 rpcz_store.cc:276] 0807 11:11:11.472399 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:12.979917 (+1507518us) service_pool.cc:227] Handling call
0807 11:11:13.012043 (+ 32126us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:13.037109 (+ 25066us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 11:11:13.065219 (+ 28110us) log.cc:842] Serialized 14 byte log entry
0807 11:11:13.896917 (+831698us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:13.896979 (+ 62us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:13.897036 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:13.897099 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:13.897209 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":28413,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":5620,"raft.queue_time_us":3496,"thread_start_us":827892,"threads_started":1}
I20260807 11:11:13.904313 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:13.903807 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 542) took 1298 ms. Trace:
I20260807 11:11:13.904987 25786 raft_consensus.cc:2454] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:11:13.906075 26041 raft_consensus.cc:1226] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 10 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 9. Current term is 10. Ops: []
I20260807 11:11:13.907996 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 541) took 1303 ms. Trace:
I20260807 11:11:13.908313 26041 rpcz_store.cc:276] 0807 11:11:12.604947 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:13.890617 (+1285670us) service_pool.cc:227] Handling call
0807 11:11:13.905936 (+ 15319us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:13.907760 (+ 1824us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:13.907946 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:13.907159 26042 rpcz_store.cc:276] 0807 11:11:12.605304 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:13.903062 (+1297758us) service_pool.cc:227] Handling call
0807 11:11:13.903357 (+ 295us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:13.903511 (+ 154us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:13.903532 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:13.903569 (+ 37us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:13.903604 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:13.903646 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:13.903730 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:13.897842 30557 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 9 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:13.897699 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:13.917611 25926 raft_consensus.cc:1226] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 5. Current term is 6. Ops: []
I20260807 11:11:13.916206 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:13.919044 26041 raft_consensus.cc:1261] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 11:11:13.908988 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:13.878638 30171 raft_consensus.cc:504] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:13.921068 26042 raft_consensus.cc:1226] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 5. Current term is 6. Ops: []
I20260807 11:11:13.921228 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:13.923256 30547 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:13.945261 25931 raft_consensus.cc:2379] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 6: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:11:13.923399 25920 raft_consensus.cc:1226] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 7. Current term is 9. Ops: []
I20260807 11:11:13.945461 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1969) took 1217 ms. Trace:
I20260807 11:11:13.929347 25981 leader_election.cc:398] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 6. Current term is 7.
I20260807 11:11:13.910414 30324 raft_consensus.cc:2726] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:13.918494 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:13.933416 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:13.954319 25796 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Advancing to term 9
W20260807 11:11:13.937923 30544 consensus_meta.cc:287] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.800s user 0.011s sys 0.000s
I20260807 11:11:13.905336 29417 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:13.938716 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1956) took 1736 ms. Trace:
I20260807 11:11:13.956734 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:13.943346 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 543) took 1337 ms. Trace:
I20260807 11:11:13.946224 25981 leader_election.cc:401] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260807 11:11:13.930084 30577 consensus_queue.cc:1046] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.010s
I20260807 11:11:13.948232 30171 leader_election.cc:288] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 8 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:13.948858 26979 raft_consensus.cc:1226] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 7. Current term is 8. Ops: []
I20260807 11:11:13.945919 26042 rpcz_store.cc:276] 0807 11:11:12.731347 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:13.910390 (+1179043us) service_pool.cc:227] Handling call
0807 11:11:13.920874 (+ 10484us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:13.944902 (+ 24028us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:13.945184 (+ 282us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:13.949307 29851 raft_consensus.cc:3037] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:13.950328 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:13.957141 26037 rpcz_store.cc:276] 0807 11:11:12.202555 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:13.102898 (+900343us) service_pool.cc:227] Handling call
0807 11:11:13.128385 (+ 25487us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:13.132682 (+ 4297us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:13.132730 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:13.163846 (+ 31116us) log.cc:842] Serialized 26 byte log entry
0807 11:11:13.861870 (+698024us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:11:13.861924 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:13.861998 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:13.938462 (+ 76464us) raft_consensus.cc:1671] finished
0807 11:11:13.938484 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:13.938648 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1867,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":5088,"thread_start_us":679680,"threads_started":1,"wal-append.queue_time_us":728971}
I20260807 11:11:13.958071 26041 rpcz_store.cc:276] 0807 11:11:12.605605 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:13.909033 (+1303428us) service_pool.cc:227] Handling call
0807 11:11:13.918724 (+ 9691us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:13.943126 (+ 24402us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:13.943298 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:13.969700 25872 leader_election.cc:302] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:13.959606 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:13.965143 25752 leader_election.cc:302] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:13.967995 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:13.958966 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:13.975238 26987 raft_consensus.cc:2454] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 8.
I20260807 11:11:13.916775 30585 consensus_queue.cc:1035] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.046s
I20260807 11:11:13.981149 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:13.977048 25872 leader_election.cc:302] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
W20260807 11:11:13.981460 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 52277.1us
I20260807 11:11:13.982016 26042 raft_consensus.cc:2454] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 7.
I20260807 11:11:13.976181 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:13.985749 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:13.987468 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:13.988580 26036 raft_consensus.cc:2379] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 8: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:11:13.963780 30544 raft_consensus.cc:504] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:13.945292 30695 consensus_queue.cc:1035] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260807 11:11:13.989624 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:14.009511 26036 raft_consensus.cc:2454] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:11:14.011126 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:14.012972 26036 raft_consensus.cc:2396] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260807 11:11:14.003006 25931 raft_consensus.cc:2454] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:11:14.023531 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:14.025607 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:14.028996 26037 raft_consensus.cc:2454] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:11:14.029094 26042 raft_consensus.cc:2454] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 7.
I20260807 11:11:14.027957 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:13.971565 30324 raft_consensus.cc:2726] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 6. Current term is 7.
I20260807 11:11:14.041541 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:14.042088 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:14.042551 26036 raft_consensus.cc:2396] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260807 11:11:14.042757 26041 raft_consensus.cc:2362] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6. Re-sending same reply.
I20260807 11:11:14.045871 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:14.048985 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:14.049855 26036 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 3.
I20260807 11:11:14.050930 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:13.996697 30544 leader_election.cc:288] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:13.986660 30415 raft_consensus.cc:2781] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:11:13.963447 30500 consensus_queue.cc:1035] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.064s
I20260807 11:11:13.964457 30557 consensus_queue.cc:1035] T 8467fdc1886745c1a7049de84b30bcc0 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
I20260807 11:11:14.076591 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:11:14.059504 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 60367.6us
I20260807 11:11:14.073786 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:14.080127 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:14.080673 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:14.075181 26938 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:14.081880 26036 raft_consensus.cc:2454] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:11:14.084031 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:14.085916 25786 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:11:14.081380 26042 raft_consensus.cc:2454] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:11:13.980191 30694 consensus_queue.cc:1035] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.085s
I20260807 11:11:14.087589 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:14.096387 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:14.099231 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:14.060928 30415 raft_consensus.cc:482] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:14.099889 26036 raft_consensus.cc:2454] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 4.
I20260807 11:11:14.104158 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:14.106864 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:14.108016 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:14.109890 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:14.111171 26041 raft_consensus.cc:2379] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 7: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:11:14.111776 29417 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:14.112288 25981 leader_election.cc:302] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:14.015156 30718 consensus_queue.cc:1046] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.114s
I20260807 11:11:14.100802 30695 raft_consensus.cc:2781] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:11:14.112658 30696 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:14.119107 25926 raft_consensus.cc:1261] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260807 11:11:14.141188 30724 consensus_queue.cc:1035] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 11:11:14.141479 30278 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:14.116510 29417 consensus_queue.cc:1046] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 11:11:14.146093 26979 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50942 (request call id 507) took 1506 ms. Trace:
I20260807 11:11:14.275744 26979 rpcz_store.cc:276] 0807 11:11:12.639240 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:12.639533 (+ 293us) service_pool.cc:227] Handling call
0807 11:11:12.690394 (+ 50861us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:13.970223 (+1279829us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:14.145764 (+175541us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b54007409d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:14.146010 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2893634944}
I20260807 11:11:14.149580 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1953) took 2049 ms. Trace:
I20260807 11:11:14.277078 26038 rpcz_store.cc:276] 0807 11:11:12.099869 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:13.091598 (+991729us) service_pool.cc:227] Handling call
0807 11:11:13.100744 (+ 9146us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:13.867119 (+766375us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:14.149208 (+282089us) spinlock_profiling.cc:243] Waited 737 ms on lock 0x7b540009ef54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:14.149483 (+ 275us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1694211456}
I20260807 11:11:14.150712 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:14.057328 30708 consensus_queue.cc:1046] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.077s
I20260807 11:11:14.157372 29847 raft_consensus.cc:3037] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
W20260807 11:11:14.161016 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 48819.3us
I20260807 11:11:14.161242 30557 raft_consensus.cc:482] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:11:14.161952 30720 log.cc:924] Time spent T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.181s user 0.000s sys 0.001s
I20260807 11:11:14.305238 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:14.305603 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:14.306177 26042 raft_consensus.cc:3037] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:14.305292 25981 leader_election.cc:302] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:11:14.202349 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:14.106716 30415 raft_consensus.cc:3037] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:11:14.208395 26036 raft_consensus.cc:1226] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:14.262084 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1951) took 2228 ms. Trace:
I20260807 11:11:14.100162 30562 consensus_queue.cc:1035] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.108s
I20260807 11:11:14.312655 26036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1970) took 1567 ms. Trace:
I20260807 11:11:14.120294 30324 raft_consensus.cc:2726] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:14.307677 25918 raft_consensus.cc:1261] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 11:11:14.204053 30695 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:14.273180 30475 raft_consensus.cc:2726] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6.
I20260807 11:11:14.312633 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:14.305966 26038 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
I20260807 11:11:14.313457 26036 rpcz_store.cc:276] 0807 11:11:12.745445 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:14.182057 (+1436612us) service_pool.cc:227] Handling call
0807 11:11:14.208122 (+ 26065us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:14.312418 (+104296us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:14.312600 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:14.310755 26039 rpcz_store.cc:276] 0807 11:11:12.033999 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:13.085292 (+1051293us) service_pool.cc:227] Handling call
0807 11:11:13.118516 (+ 33224us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:13.133309 (+ 14793us) raft_consensus.cc:1520] Early marking committed up to index 7
0807 11:11:13.187156 (+ 53847us) log.cc:842] Serialized 14 byte log entry
0807 11:11:14.261538 (+1074382us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:14.261625 (+ 87us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:14.261699 (+ 74us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:14.261807 (+ 108us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:14.261997 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4017,"prepare.queue_time_us":53619,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":1604,"raft.queue_time_us":852,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":4692,"thread_start_us":1059019,"threads_started":2,"wal-append.queue_time_us":705056}
I20260807 11:11:14.327816 25796 raft_consensus.cc:2454] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 9.
I20260807 11:11:14.283643 30718 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:14.317199 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:14.323678 25921 raft_consensus.cc:1261] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 11:11:14.330214 25925 raft_consensus.cc:2379] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 5: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
W20260807 11:11:14.329999 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 289474us
I20260807 11:11:14.332573 26938 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:11:14.335537 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 360045us
I20260807 11:11:14.338449 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:14.338701 25872 leader_election.cc:302] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:14.340477 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:14.344276 25797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 432) took 1143 ms. Trace:
W20260807 11:11:14.382478 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 50049.3us
I20260807 11:11:14.386761 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1881) took 1411 ms. Trace:
I20260807 11:11:14.386698 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:14.387264 26039 rpcz_store.cc:276] 0807 11:11:12.975034 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:14.325926 (+1350892us) service_pool.cc:227] Handling call
0807 11:11:14.386112 (+ 60186us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:14.386379 (+ 267us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:14.386413 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:14.386464 (+ 51us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:14.386514 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:14.386579 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:14.386696 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:14.388558 25921 raft_consensus.cc:2379] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 9: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:11:14.322860 30695 raft_consensus.cc:3037] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:11:14.394246 26936 leader_election.cc:302] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:14.390187 30324 raft_consensus.cc:482] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:14.309458 30557 raft_consensus.cc:504] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:14.413964 26042 raft_consensus.cc:2454] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 8.
W20260807 11:11:14.409428 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 504494us
I20260807 11:11:14.448508 26938 leader_election.cc:302] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:14.410925 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1883) took 1289 ms. Trace:
I20260807 11:11:14.449369 26039 rpcz_store.cc:276] 0807 11:11:13.121245 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:14.389169 (+1267924us) service_pool.cc:227] Handling call
0807 11:11:14.410372 (+ 21203us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:14.410546 (+ 174us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:14.410581 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:14.410631 (+ 50us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:14.410679 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:14.410742 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:14.410857 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:14.408321 25796 raft_consensus.cc:1261] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260807 11:11:14.415032 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:14.435868 26036 raft_consensus.cc:1226] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 5. Current term is 6. Ops: []
I20260807 11:11:14.385035 30544 raft_consensus.cc:2726] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:11:14.440136 25981 leader_election.cc:398] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 6. Current term is 7.
I20260807 11:11:14.469025 25981 leader_election.cc:401] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
W20260807 11:11:14.469735 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 59298.8us
I20260807 11:11:14.469877 30730 raft_consensus.cc:2726] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 6. Current term is 7.
I20260807 11:11:14.470709 25981 leader_election.cc:398] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6.
I20260807 11:11:14.471067 25981 leader_election.cc:401] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260807 11:11:14.471726 30730 raft_consensus.cc:2726] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6.
I20260807 11:11:14.392652 30700 raft_consensus.cc:2781] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:11:14.473040 30700 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:14.473497 30700 raft_consensus.cc:3037] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:11:14.472466 25981 leader_election.cc:302] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:11:14.476222 30730 raft_consensus.cc:2781] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 11:11:14.483711 30700 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:14.491894 30700 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:11:14.439666 30320 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.021s sys 0.031s
I20260807 11:11:14.411787 30630 raft_consensus.cc:2726] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260807 11:11:14.502183 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 53797.8us
I20260807 11:11:14.439213 29966 consensus_queue.cc:1035] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.103s
W20260807 11:11:14.501533 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 51673.1us
I20260807 11:11:14.505556 25931 raft_consensus.cc:1261] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260807 11:11:14.506138 25916 raft_consensus.cc:1226] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260807 11:11:14.382740 30718 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:14.517695 25801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 437) took 1312 ms. Trace:
I20260807 11:11:14.522691 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:14.602882 25918 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Advancing to term 9
I20260807 11:11:14.449995 29991 raft_consensus.cc:2741] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260807 11:11:14.409024 30324 raft_consensus.cc:504] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:14.570768 25798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1889) took 1934 ms. Trace:
W20260807 11:11:14.571317 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 98920.9us
W20260807 11:11:14.567493 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 109349us
I20260807 11:11:14.469044 26036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1882) took 1435 ms. Trace:
I20260807 11:11:14.512825 30724 raft_consensus.cc:2781] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 11:11:14.602456 26042 raft_consensus.cc:1226] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 6. Current term is 7. Ops: []
W20260807 11:11:14.581139 30320 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.667s user 0.022s sys 0.031s
I20260807 11:11:14.609061 26036 rpcz_store.cc:276] 0807 11:11:13.033286 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:14.350007 (+1316721us) service_pool.cc:227] Handling call
0807 11:11:14.435619 (+ 85612us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:14.468692 (+ 33073us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:14.468969 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:14.610451 25750 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:14.602258 25801 rpcz_store.cc:276] 0807 11:11:13.205548 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:13.207821 (+ 2273us) service_pool.cc:227] Handling call
0807 11:11:13.220941 (+ 13120us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:13.755658 (+534717us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:14.517331 (+761673us) spinlock_profiling.cc:243] Waited 534 ms on lock 0x7b5400046fd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:14.517607 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1228545280}
I20260807 11:11:14.608071 25798 rpcz_store.cc:276] 0807 11:11:12.635972 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:13.739609 (+1103637us) service_pool.cc:227] Handling call
0807 11:11:13.740013 (+ 404us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:13.740395 (+ 382us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:13.740423 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:13.740971 (+ 548us) log.cc:842] Serialized 26 byte log entry
0807 11:11:13.895059 (+154088us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:13.895197 (+ 138us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:13.895265 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:14.570503 (+675238us) raft_consensus.cc:1671] finished
0807 11:11:14.570532 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:14.570692 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1281,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":123,"thread_start_us":153585,"threads_started":1,"wal-append.queue_time_us":213026}
I20260807 11:11:14.623020 25803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 434) took 1421 ms. Trace:
I20260807 11:11:14.623896 25803 rpcz_store.cc:276] 0807 11:11:13.201648 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:13.208500 (+ 6852us) service_pool.cc:227] Handling call
0807 11:11:13.235412 (+ 26912us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:13.235669 (+ 257us) raft_consensus.cc:1520] Early marking committed up to index 9
0807 11:11:14.220909 (+985240us) log.cc:842] Serialized 14 byte log entry
0807 11:11:14.497303 (+276394us) log.cc:842] Serialized 14 byte log entry
0807 11:11:14.620624 (+123321us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:14.620683 (+ 59us) raft_consensus.cc:1631] Marking committed up to 9
0807 11:11:14.620736 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:14.620840 (+ 104us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:14.622954 (+ 2114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":540707,"prepare.queue_time_us":646169,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":74,"raft.queue_time_us":235365,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":19674,"thread_start_us":748934,"threads_started":3}
I20260807 11:11:14.628455 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1884) took 1494 ms. Trace:
I20260807 11:11:14.629001 26042 rpcz_store.cc:276] 0807 11:11:13.134072 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:14.598699 (+1464627us) service_pool.cc:227] Handling call
0807 11:11:14.599119 (+ 420us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:14.628139 (+ 29020us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:14.628395 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:14.621510 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:14.518790 28964 consensus_queue.cc:1035] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.090s
I20260807 11:11:14.641417 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:14.641583 26036 raft_consensus.cc:3037] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 5
I20260807 11:11:14.647732 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:14.627876 30278 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 10 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:14.658282 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:14.659050 25801 raft_consensus.cc:3037] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
W20260807 11:11:14.660960 25875 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.036s sys 0.058s
I20260807 11:11:14.667112 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1965) took 2208 ms. Trace:
I20260807 11:11:14.669878 26042 raft_consensus.cc:1261] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20260807 11:11:14.664034 30256 consensus_meta.cc:287] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.921s user 0.012s sys 0.000s
W20260807 11:11:14.662200 29851 consensus_meta.cc:287] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.682s user 0.010s sys 0.002s
I20260807 11:11:14.670799 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:14.702193 25916 raft_consensus.cc:2379] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 5: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:11:14.672693 25803 raft_consensus.cc:1226] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260807 11:11:14.666671 30557 leader_election.cc:288] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:11:14.682512 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 237992us
W20260807 11:11:14.666639 30421 consensus_meta.cc:287] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.916s user 0.010s sys 0.000s
I20260807 11:11:14.684072 26033 rpcz_store.cc:276] 0807 11:11:12.458623 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:13.808543 (+1349920us) service_pool.cc:227] Handling call
0807 11:11:13.832554 (+ 24011us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:13.886674 (+ 54120us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:13.886718 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:13.902656 (+ 15938us) log.cc:842] Serialized 26 byte log entry
0807 11:11:14.435935 (+533279us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:14.435987 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:14.436078 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:14.666873 (+230795us) raft_consensus.cc:1671] finished
0807 11:11:14.666893 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:14.667036 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5760,"raft.queue_time_us":13660,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":7721,"thread_start_us":497398,"threads_started":1,"wal-append.queue_time_us":751679}
I20260807 11:11:14.646468 30730 raft_consensus.cc:686] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:11:14.687280 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 547) took 1486 ms. Trace:
I20260807 11:11:14.646405 30700 consensus_queue.cc:1035] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
W20260807 11:11:14.675966 30398 consensus_meta.cc:287] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.940s user 0.013s sys 0.001s
I20260807 11:11:14.645910 30724 raft_consensus.cc:686] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:14.696782 25808 raft_consensus.cc:1226] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:14.694226 25918 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 9.
I20260807 11:11:14.708200 26042 rpcz_store.cc:276] 0807 11:11:13.200922 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:14.629863 (+1428941us) service_pool.cc:227] Handling call
0807 11:11:14.664525 (+ 34662us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:14.687006 (+ 22481us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:14.687225 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:11:14.683393 25875 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.792s user 0.036s sys 0.061s
I20260807 11:11:14.708565 25916 raft_consensus.cc:1226] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 10 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
W20260807 11:11:14.649977 25869 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.005s sys 0.022s
W20260807 11:11:14.710618 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 69142.5us
I20260807 11:11:14.688886 30278 raft_consensus.cc:2726] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260807 11:11:14.718968 25869 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.638s user 0.005s sys 0.023s
W20260807 11:11:14.719692 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 638824us
I20260807 11:11:14.726110 25916 raft_consensus.cc:1261] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 11:11:14.684211 30694 consensus_queue.cc:1046] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.031s
I20260807 11:11:14.706678 30666 consensus_queue.cc:1035] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260807 11:11:14.701301 30654 consensus_queue.cc:1035] T 9c6408e9acee46adacd2bbd81669f27a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.024s
I20260807 11:11:14.710012 30517 raft_consensus.cc:3032] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 11:11:14.712558 30582 consensus_queue.cc:1046] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.022s
I20260807 11:11:14.700186 30256 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:14.716231 30696 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:11:14.720973 30751 log.cc:924] Time spent T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.115s user 0.001s sys 0.000s
I20260807 11:11:14.726300 30756 raft_consensus.cc:2781] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:14.748145 30756 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:14.748602 30756 raft_consensus.cc:3037] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:14.728061 25916 pending_rounds.cc:76] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc: Aborting all ops after (but not including) 7
I20260807 11:11:14.754580 25916 pending_rounds.cc:97] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260807 11:11:14.754616 25869 leader_election.cc:302] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:14.754930 25916 raft_consensus.cc:2866] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 11:11:14.742820 25917 raft_consensus.cc:2379] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 8: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:11:14.733995 25981 leader_election.cc:302] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:14.758152 26036 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:11:14.761349 26036 raft_consensus.cc:1261] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260807 11:11:14.764508 25801 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:11:14.764756 26036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 550) took 1561 ms. Trace:
I20260807 11:11:14.765633 26985 raft_consensus.cc:1261] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 10 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260807 11:11:14.766903 26985 pending_rounds.cc:76] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543: Aborting all ops after (but not including) 8
I20260807 11:11:14.767180 26985 pending_rounds.cc:97] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260807 11:11:14.767589 26985 raft_consensus.cc:2866] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260807 11:11:14.731581 30754 log.cc:924] Time spent T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.119s user 0.000s sys 0.000s
I20260807 11:11:14.779779 25869 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:11:14.743939 30256 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:14.781239 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
W20260807 11:11:14.767073 30270 consensus_meta.cc:287] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 1.047s user 0.012s sys 0.001s
I20260807 11:11:14.766799 26036 rpcz_store.cc:276] 0807 11:11:13.203518 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:14.760020 (+1556502us) service_pool.cc:227] Handling call
0807 11:11:14.760346 (+ 326us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:14.763431 (+ 3085us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:14.763654 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:14.762799 30743 raft_consensus.cc:2781] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 11:11:14.790362 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:14.734570 30517 raft_consensus.cc:729] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:14.712395 30421 raft_consensus.cc:504] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:14.733192 30718 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:14.715647 30390 consensus_queue.cc:1046] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.057s
I20260807 11:11:14.773864 29692 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:14.727770 30730 consensus_queue.cc:237] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:14.741762 30398 consensus_queue.cc:1035] T 0fc5b1dfff1a4556abdd562afbebd768 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20260807 11:11:14.808490 29692 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:14.740293 29851 consensus_queue.cc:1046] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20260807 11:11:14.810120 30398 raft_consensus.cc:3032] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 LEADER]: Stepping down as leader of term 9
I20260807 11:11:14.810573 30398 raft_consensus.cc:729] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:14.739161 30695 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:14.797823 30582 consensus_queue.cc:1035] T adab2949574e4c6badfcf68e51b9c072 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260807 11:11:14.796940 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 458303us
I20260807 11:11:14.798398 26036 raft_consensus.cc:1226] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 10 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260807 11:11:14.768793 30776 raft_consensus.cc:2726] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260807 11:11:14.802862 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 155736us
I20260807 11:11:14.802845 25981 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:14.806455 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:14.744990 30547 consensus_queue.cc:1035] T 51fb619c2ada475f817fbcb9739a401a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.045s
I20260807 11:11:14.811714 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:14.734097 30324 leader_election.cc:288] T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:14.806910 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:14.796113 25869 leader_election.cc:302] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:14.814570 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:14.805653 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 546) took 1606 ms. Trace:
I20260807 11:11:14.824285 26036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 551) took 1028 ms. Trace:
I20260807 11:11:14.825651 25872 leader_election.cc:302] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:11:14.801373 30556 raft_consensus.cc:3032] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 LEADER]: Stepping down as leader of term 8
I20260807 11:11:14.802205 30666 raft_consensus.cc:3032] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 11:11:14.794946 30278 raft_consensus.cc:2781] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260807 11:11:14.849288 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 142661us
W20260807 11:11:14.946894 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 155870us
I20260807 11:11:14.947430 25752 leader_election.cc:302] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:14.912608 26036 rpcz_store.cc:276] 0807 11:11:13.796080 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:14.797485 (+1001405us) service_pool.cc:227] Handling call
0807 11:11:14.798079 (+ 594us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:14.823885 (+ 25806us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:14.824187 (+ 302us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:14.946550 30556 raft_consensus.cc:729] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:14.952077 30556 consensus_queue.cc:260] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:14.954167 30556 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Advancing to term 9
I20260807 11:11:14.916381 25920 raft_consensus.cc:2379] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 7: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
W20260807 11:11:14.916263 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 120246us
I20260807 11:11:14.957826 25869 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:11:14.912431 26039 rpcz_store.cc:276] 0807 11:11:13.198028 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:14.598208 (+1400180us) service_pool.cc:227] Handling call
0807 11:11:14.632211 (+ 34003us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:14.632495 (+ 284us) raft_consensus.cc:1520] Early marking committed up to index 6
0807 11:11:14.676466 (+ 43971us) log.cc:842] Serialized 14 byte log entry
0807 11:11:14.803665 (+127199us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:14.803731 (+ 66us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:14.803788 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:14.803870 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:14.804002 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":43802,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":629,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":5664,"thread_start_us":97571,"threads_started":1,"wal-append.queue_time_us":133484}
I20260807 11:11:14.773741 29915 consensus_queue.cc:1046] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.046s
I20260807 11:11:14.897801 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1957) took 2694 ms. Trace:
I20260807 11:11:14.912680 25801 raft_consensus.cc:2379] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 5: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:11:14.804939 30270 raft_consensus.cc:2726] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:14.805059 30743 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:14.753355 30724 consensus_queue.cc:237] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:14.843065 26979 raft_consensus.cc:2379] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 5: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:11:14.949651 30320 raft_consensus.cc:2781] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:14.962600 26035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1960) took 2758 ms. Trace:
I20260807 11:11:14.966724 25869 leader_election.cc:302] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:14.968886 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:14.970280 25752 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:14.970502 30743 raft_consensus.cc:3037] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Advancing to term 9
I20260807 11:11:14.950700 30278 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:14.967880 30557 raft_consensus.cc:2781] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Leader election won for term 9
I20260807 11:11:14.973017 29851 raft_consensus.cc:2781] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260807 11:11:14.976308 30762 log.cc:924] Time spent T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.255s user 0.000s sys 0.000s
I20260807 11:11:14.949733 30726 raft_consensus.cc:2781] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260807 11:11:14.964231 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 138698us
I20260807 11:11:15.013468 25872 leader_election.cc:302] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:11:14.837211 30741 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:14.962816 26046 rpcz_store.cc:276] 0807 11:11:12.203078 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:13.752368 (+1549290us) service_pool.cc:227] Handling call
0807 11:11:13.752662 (+ 294us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:13.808648 (+ 55986us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 11:11:14.382191 (+573543us) log.cc:842] Serialized 14 byte log entry
0807 11:11:14.816815 (+434624us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:14.816889 (+ 74us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:14.897369 (+ 80480us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:14.897514 (+ 145us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:14.897725 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56833,"prepare.queue_time_us":107696,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":48,"raft.queue_time_us":32657,"raft.run_cpu_time_us":61,"raft.run_wall_time_us":61,"thread_start_us":864059,"threads_started":3,"wal-append.queue_time_us":391017}
W20260807 11:11:14.986054 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 144701us
I20260807 11:11:15.018200 25803 raft_consensus.cc:1226] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260807 11:11:14.986989 25981 leader_election.cc:302] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:14.950009 30666 raft_consensus.cc:729] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:14.949815 30767 raft_consensus.cc:2726] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:14.995796 25932 raft_consensus.cc:2454] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 7.
I20260807 11:11:14.999234 25869 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:15.030481 25872 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:15.000521 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:15.008071 25752 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:15.012324 30794 raft_consensus.cc:2781] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:14.996958 30320 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 11:11:15.037374 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 50473.4us
I20260807 11:11:15.032039 25801 raft_consensus.cc:3037] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:15.001783 30278 raft_consensus.cc:3037] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:15.008221 29851 raft_consensus.cc:482] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:15.001988 30256 raft_consensus.cc:2726] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260807 11:11:15.057559 30256 raft_consensus.cc:2726] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 11:11:15.057431 30666 consensus_queue.cc:260] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:15.058419 30256 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Advancing to term 10
I20260807 11:11:15.058730 25920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1841) took 1080 ms. Trace:
I20260807 11:11:15.059041 30666 raft_consensus.cc:3037] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:14.996896 30324 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:15.061405 30771 raft_consensus.cc:3032] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 11:11:15.030107 25655 catalog_manager.cc:5495] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:11:15.061825 30771 raft_consensus.cc:729] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:15.061980 25752 leader_election.cc:302] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:15.060583 25806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1909) took 1149 ms. Trace:
I20260807 11:11:15.061427 30324 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:15.062636 25806 rpcz_store.cc:276] 0807 11:11:13.911519 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:13.911672 (+ 153us) service_pool.cc:227] Handling call
0807 11:11:13.919846 (+ 8174us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:13.920094 (+ 248us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 11:11:13.959565 (+ 39471us) log.cc:842] Serialized 14 byte log entry
0807 11:11:13.959970 (+ 405us) log.cc:842] Serialized 14 byte log entry
0807 11:11:14.520319 (+560349us) spinlock_profiling.cc:243] Waited 2.79 ms on lock 0x7b18002ff380. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbc79c638 00007f3fbc79fb11 00007f3fbc79fac9 00007f3fbc79fa59 00007f3fbc79fa21 00007f3fbc79ed1d 00007f3fc5a00204 00007f3fc5a00039 00007f3fbc77edc6
0807 11:11:15.060004 (+539685us) spinlock_profiling.cc:243] Waited 1.41 ms on lock 0x7b18002ff380. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbc7134ca 00007f3fc5df4f99 00007f3fbf361e10 00007f3fbf355663 00007f3fbf34b02a 00007f3fbf34b57b 00007f3fbf6ce88a 00007f3fbf6ddf0f 00007f3fbf6dde88
0807 11:11:15.060229 (+ 225us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:15.060280 (+ 51us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:15.060335 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:15.060412 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:15.060521 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1492,"prepare.queue_time_us":600637,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":75,"raft.queue_time_us":2799,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":4453,"spinlock_wait_cycles":9658112}
I20260807 11:11:15.065274 29851 raft_consensus.cc:3037] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:15.035706 30794 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:15.065958 30794 raft_consensus.cc:3037] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:15.059098 25920 rpcz_store.cc:276] 0807 11:11:13.978201 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:13.978370 (+ 169us) service_pool.cc:227] Handling call
0807 11:11:15.058516 (+1080146us) spinlock_profiling.cc:243] Waited 930 ms on lock 0x7b54001806d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:15.058670 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2138073856}
I20260807 11:11:15.029924 30741 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:15.024545 30557 raft_consensus.cc:686] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [term 9 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:15.067924 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:15.042774 30767 raft_consensus.cc:2726] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 11:11:15.069020 30557 consensus_queue.cc:237] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:15.036105 26046 raft_consensus.cc:1226] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:14.835758 30398 consensus_queue.cc:260] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:15.093384 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1903) took 1127 ms. Trace:
I20260807 11:11:13.835134 25800 rpcz_store.cc:276] 0807 11:11:12.638385 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:12.825558 (+187173us) service_pool.cc:227] Handling call
0807 11:11:12.917885 (+ 92327us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:11:12.925486 (+ 7601us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:12.925532 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 11:11:12.958200 (+ 32668us) log.cc:842] Serialized 52 byte log entry
0807 11:11:13.127382 (+169182us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:13.422510 (+295128us) log.cc:842] Serialized 14 byte log entry
0807 11:11:13.815376 (+392866us) log.cc:842] Serialized 14 byte log entry
0807 11:11:13.815824 (+ 448us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:13.815939 (+ 115us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:13.816006 (+ 67us) raft_consensus.cc:1671] finished
0807 11:11:13.816017 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:13.816210 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":399220,"raft.queue_time_us":16724,"raft.run_cpu_time_us":485,"raft.run_wall_time_us":49566,"thread_start_us":389671,"threads_started":2,"wal-append.queue_time_us":240787}
W20260807 11:11:15.041574 25750 connection.cc:232] RPC call timeout handler was delayed by -1.1204628560226411s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19920000000000002s for a response
I20260807 11:11:15.012775 30726 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:15.047225 28964 raft_consensus.cc:2781] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 11:11:15.047679 30421 leader_election.cc:288] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:15.050019 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:14.968549 30517 consensus_queue.cc:260] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:15.120615 30517 raft_consensus.cc:3037] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:15.023335 30758 raft_consensus.cc:482] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:15.125116 30758 raft_consensus.cc:504] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:15.054281 30696 raft_consensus.cc:2781] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:11:15.113003 30693 consensus_queue.cc:1046] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20260807 11:11:15.087741 30695 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:15.129036 30758 leader_election.cc:288] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:11:15.131254 30415 consensus_meta.cc:287] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.811s user 0.008s sys 0.004s
I20260807 11:11:15.095929 26046 rpcz_store.cc:276] 0807 11:11:13.966218 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:15.017127 (+1050909us) service_pool.cc:227] Handling call
0807 11:11:15.035897 (+ 18770us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:15.093047 (+ 57150us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:15.093315 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:15.131690 30415 raft_consensus.cc:504] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:15.132825 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:15.133680 26046 raft_consensus.cc:3037] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
W20260807 11:11:15.133924 29847 consensus_meta.cc:287] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.830s user 0.012s sys 0.000s
I20260807 11:11:15.134191 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:15.033262 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 548) took 1830 ms. Trace:
W20260807 11:11:15.095227 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 56987.2us
W20260807 11:11:15.096784 25750 connection.cc:232] RPC call timeout handler was delayed by -1.0898628558497876s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2026s for a response
I20260807 11:11:15.135722 26033 rpcz_store.cc:276] 0807 11:11:13.202778 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:14.707545 (+1504767us) service_pool.cc:227] Handling call
0807 11:11:14.718157 (+ 10612us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:14.718395 (+ 238us) raft_consensus.cc:1520] Early marking committed up to index 8
0807 11:11:14.753271 (+ 34876us) log.cc:842] Serialized 14 byte log entry
0807 11:11:15.032891 (+279620us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:15.032948 (+ 57us) raft_consensus.cc:1631] Marking committed up to 8
0807 11:11:15.033000 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:15.033068 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:15.033205 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":34398,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":37000,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":76,"thread_start_us":279249,"threads_started":1,"wal-append.queue_time_us":364489}
I20260807 11:11:15.062788 30771 consensus_queue.cc:260] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:15.119850 30324 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:15.119989 30544 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:15.126673 30696 raft_consensus.cc:686] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:15.085739 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:15.142804 25806 raft_consensus.cc:3037] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:15.134378 29847 raft_consensus.cc:2726] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:15.039952 30320 raft_consensus.cc:3037] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:15.109236 26985 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 514) took 2907 ms. Trace:
W20260807 11:11:15.136636 25750 connection.cc:232] RPC call timeout handler was delayed by -1.0745628559961915s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20430000000000004s for a response
W20260807 11:11:15.112317 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 50408.1us
W20260807 11:11:15.112236 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 81864.2us
W20260807 11:11:15.153939 25750 connection.cc:232] RPC call timeout handler was delayed by -1.051162855932489s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2069s for a response
I20260807 11:11:15.153723 26985 rpcz_store.cc:276] 0807 11:11:12.169102 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:12.169265 (+ 163us) service_pool.cc:227] Handling call
0807 11:11:12.188280 (+ 19015us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:14.773696 (+2585416us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:15.042702 (+269006us) spinlock_profiling.cc:243] Waited 2.53 s on lock 0x7b540072f454. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:15.076527 (+ 33825us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5824879872}
W20260807 11:11:15.154556 25750 connection.cc:232] RPC call timeout handler was delayed by -1.0187628560233861s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21050000000000002s for a response
I20260807 11:11:15.155282 25872 leader_election.cc:302] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:11:15.156394 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:11:15.141934 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 434883us
I20260807 11:11:15.132893 25808 raft_consensus.cc:2379] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 5: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:11:15.114363 30726 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Advancing to term 9
I20260807 11:11:15.009202 30756 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:15.156594 25872 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:15.173754 25926 raft_consensus.cc:1226] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:15.163823 25801 raft_consensus.cc:2454] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 8.
I20260807 11:11:15.114226 28964 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:15.188112 25981 leader_election.cc:302] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:15.188958 25752 leader_election.cc:302] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:15.188542 25872 leader_election.cc:302] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:15.192144 25872 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:15.192318 25869 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:11:15.181354 26936 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:15.193681 25752 leader_election.cc:302] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:15.179318 25750 leader_election.cc:302] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:15.199265 26046 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:11:15.201882 25752 leader_election.cc:302] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:15.201884 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 553) took 1079 ms. Trace:
I20260807 11:11:15.140475 30795 raft_consensus.cc:2781] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 11:11:15.158468 30770 raft_consensus.cc:482] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:15.173933 30780 raft_consensus.cc:482] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:15.213260 25872 leader_election.cc:302] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:15.200560 30820 raft_consensus.cc:2781] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 11:11:15.206496 26046 rpcz_store.cc:276] 0807 11:11:14.122469 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:15.201131 (+1078662us) service_pool.cc:227] Handling call
0807 11:11:15.201427 (+ 296us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:15.201589 (+ 162us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:15.201625 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:15.201663 (+ 38us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:15.201701 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:15.201749 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:15.201836 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:15.200273 28964 raft_consensus.cc:3037] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:15.208554 30730 raft_consensus.cc:2726] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260807 11:11:15.225858 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 50551.1us
I20260807 11:11:15.228201 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 555) took 1087 ms. Trace:
I20260807 11:11:15.153225 30544 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:15.230994 29915 raft_consensus.cc:2726] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 11:11:15.205051 30718 raft_consensus.cc:2781] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:11:15.240979 26046 rpcz_store.cc:276] 0807 11:11:14.140647 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:15.227215 (+1086568us) service_pool.cc:227] Handling call
0807 11:11:15.227569 (+ 354us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:15.227813 (+ 244us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:15.227847 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:15.227901 (+ 54us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:15.227950 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:15.228015 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:15.228138 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:15.207631 30695 raft_consensus.cc:2726] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:14.382081 25797 rpcz_store.cc:276] 0807 11:11:13.200967 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:13.207710 (+ 6743us) service_pool.cc:227] Handling call
0807 11:11:13.221026 (+ 13316us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:13.221224 (+ 198us) raft_consensus.cc:1520] Early marking committed up to index 6
0807 11:11:14.057236 (+836012us) log.cc:842] Serialized 14 byte log entry
0807 11:11:14.343864 (+286628us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:14.343931 (+ 67us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:14.343996 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:14.344074 (+ 78us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:14.344211 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26380,"prepare.queue_time_us":660279,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":42,"raft.queue_time_us":66468,"thread_start_us":538415,"threads_started":2}
W20260807 11:11:15.251381 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 49541.3us
I20260807 11:11:15.208177 30771 raft_consensus.cc:3037] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:15.205776 30769 raft_consensus.cc:2726] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 11:11:15.226884 30820 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:15.299582 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:15.261734 25981 leader_election.cc:302] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:15.305509 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:15.304020 26046 raft_consensus.cc:2362] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 7. Re-sending same reply.
I20260807 11:11:15.309130 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:15.225203 30694 consensus_queue.cc:1046] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.072s
W20260807 11:11:15.313252 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 120989us
I20260807 11:11:15.253172 30773 raft_consensus.cc:2781] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 11:11:15.347615 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:15.248111 30718 raft_consensus.cc:482] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:15.250908 30696 consensus_queue.cc:237] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:15.350943 26046 raft_consensus.cc:2454] T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 8.
I20260807 11:11:15.389029 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:11:15.336006 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 134523us
I20260807 11:11:15.297065 29915 raft_consensus.cc:2781] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Leader election won for term 8
I20260807 11:11:15.390600 25750 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:15.389649 26046 raft_consensus.cc:2454] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 7.
I20260807 11:11:15.253623 30400 consensus_queue.cc:1046] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.094s
I20260807 11:11:15.307291 30820 raft_consensus.cc:3037] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:15.403065 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:15.314014 30256 raft_consensus.cc:2726] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 11:11:15.259907 30741 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:15.407145 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:15.411417 25869 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:15.412665 29915 raft_consensus.cc:686] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 8 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:15.400125 30630 raft_consensus.cc:3032] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 LEADER]: Stepping down as leader of term 9
I20260807 11:11:15.395766 30718 raft_consensus.cc:3037] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
W20260807 11:11:15.414069 30830 log.cc:924] Time spent T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
W20260807 11:11:15.437937 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 176292us
I20260807 11:11:15.424176 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1880) took 2465 ms. Trace:
I20260807 11:11:15.395506 30773 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:15.240263 30780 raft_consensus.cc:504] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:15.248248 30770 raft_consensus.cc:504] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:15.403059 30324 raft_consensus.cc:2726] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260807 11:11:15.448782 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 188226us
I20260807 11:11:15.435043 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:15.403440 30415 leader_election.cc:288] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:15.408970 30795 raft_consensus.cc:686] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:15.452136 30415 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:15.296620 30811 consensus_queue.cc:1035] T cf86a21774c34f33a6540d1dad0de0c4 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.170s
I20260807 11:11:15.443449 25981 leader_election.cc:302] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:15.429140 30517 raft_consensus.cc:2726] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:11:15.313480 30769 consensus_queue.cc:1035] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.119s
I20260807 11:11:15.425973 30256 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:15.459239 26936 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:11:15.439402 26038 rpcz_store.cc:276] 0807 11:11:12.958624 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:14.324328 (+1365704us) service_pool.cc:227] Handling call
0807 11:11:14.324666 (+ 338us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:14.325427 (+ 761us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 11:11:14.390741 (+ 65314us) log.cc:842] Serialized 14 byte log entry
0807 11:11:14.595373 (+204632us) log.cc:842] Serialized 14 byte log entry
0807 11:11:15.324588 (+729215us) spinlock_profiling.cc:243] Waited 6.6 ms on lock 0x7b1800102e80. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbc77e345 00007f3fc55b5a39 00007f3fbc79b8bb 00007f3fbc799cbf 00007f3fbc799155 00007f3fbf6bc3d0 00007f3fbf6ce576 00007f3fbf6df0d7 00007f3fbf6df058
0807 11:11:15.413044 (+ 88456us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:15.413181 (+ 137us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:15.423843 (+ 10662us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:15.423944 (+ 101us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:15.424093 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":65778,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":89,"raft.queue_time_us":278443,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":180,"spinlock_wait_cycles":15171456,"thread_start_us":291985,"threads_started":3,"wal-append.queue_time_us":266342}
I20260807 11:11:15.429715 30630 raft_consensus.cc:729] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:15.452255 30795 consensus_queue.cc:237] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:15.452540 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:11:15.470965 30803 log.cc:924] Time spent T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.244s user 0.001s sys 0.000s
I20260807 11:11:15.473338 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:15.444214 30773 raft_consensus.cc:3037] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:15.477104 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:15.490901 26038 raft_consensus.cc:2379] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 5: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:11:15.478116 26046 raft_consensus.cc:3037] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:15.491138 25981 leader_election.cc:302] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:15.458772 30475 raft_consensus.cc:2726] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:15.501533 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:15.452729 30400 consensus_queue.cc:1046] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.027s
I20260807 11:11:15.452661 30415 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:15.502336 26037 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:11:15.471020 30852 raft_consensus.cc:482] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:15.476402 30769 consensus_queue.cc:1225] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer 4fd6381544574ddca559ce63b9243efc log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260807 11:11:15.493240 30724 raft_consensus.cc:2781] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:11:15.490960 30390 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:11:15.564146 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 73086.8us
I20260807 11:11:15.506989 30324 raft_consensus.cc:2726] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:11:15.596062 26046 raft_consensus.cc:2454] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 7.
I20260807 11:11:15.492899 30842 raft_consensus.cc:482] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:15.501099 30557 raft_consensus.cc:2726] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:15.495329 30708 raft_consensus.cc:2726] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 11:11:15.494766 30756 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:15.601878 26037 raft_consensus.cc:1226] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 10 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 9. Current term is 10. Ops: []
I20260807 11:11:15.488780 30256 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:11:15.748855 25921 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.054s sys 0.129s
I20260807 11:11:15.750401 25872 leader_election.cc:302] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:15.564729 30544 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:11:15.566912 30807 log.cc:924] Time spent T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.164s user 0.000s sys 0.001s
I20260807 11:11:15.604656 30171 raft_consensus.cc:3032] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 LEADER]: Stepping down as leader of term 6
W20260807 11:11:15.754854 30818 log.cc:924] Time spent T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.318s user 0.000s sys 0.001s
W20260807 11:11:15.614873 30743 consensus_meta.cc:287] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.591s user 0.011s sys 0.000s
I20260807 11:11:15.766523 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 561) took 1060 ms. Trace:
I20260807 11:11:15.779737 26037 rpcz_store.cc:276] 0807 11:11:14.705799 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:15.589697 (+883898us) service_pool.cc:227] Handling call
0807 11:11:15.590826 (+ 1129us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:15.766211 (+175385us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:15.766450 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:15.781009 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:15.781721 26037 raft_consensus.cc:2454] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:11:15.563349 30278 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:15.775619 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:15.788053 25919 raft_consensus.cc:2379] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 5: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:11:15.788501 25919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35276 (request call id 389) took 1200 ms. Trace:
I20260807 11:11:15.788506 25872 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:15.788947 25919 rpcz_store.cc:276] 0807 11:11:14.588432 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:14.663865 (+ 75433us) service_pool.cc:227] Handling call
0807 11:11:15.775554 (+1111689us) spinlock_profiling.cc:243] Waited 5.38 ms on lock 0x7b5c00075b0c. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc300b361 00007f3fc31a51f9 00007f3fc4605ac6 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:15.788440 (+ 12886us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12375424}
I20260807 11:11:15.748653 30475 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
W20260807 11:11:15.777969 30833 log.cc:924] Time spent T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.320s user 0.000s sys 0.001s
I20260807 11:11:15.592711 30517 consensus_queue.cc:1035] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.164s
I20260807 11:11:15.782644 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:11:15.758863 25921 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.581s user 0.055s sys 0.129s
I20260807 11:11:15.790091 25872 leader_election.cc:302] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:15.756830 30724 raft_consensus.cc:686] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:15.802079 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:11:15.803795 25917 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1847) took 1638 ms (client timeout 1992 ms). Trace:
I20260807 11:11:15.803645 30475 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:11:15.804226 25917 rpcz_store.cc:269] 0807 11:11:14.165634 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:14.165828 (+ 194us) service_pool.cc:227] Handling call
0807 11:11:15.803599 (+1637771us) spinlock_profiling.cc:243] Waited 435 ms on lock 0x7b54003395d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:15.803737 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1001445632}
I20260807 11:11:15.805949 30475 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:11:15.806156 30746 log.cc:924] Time spent T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 1.178s user 0.000s sys 0.013s
I20260807 11:11:15.807134 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 552) took 1685 ms. Trace:
I20260807 11:11:15.807658 25796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 449) took 1692 ms. Trace:
I20260807 11:11:15.807904 25786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 443) took 1710 ms. Trace:
I20260807 11:11:15.808250 25786 rpcz_store.cc:276] 0807 11:11:14.097385 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:14.102340 (+ 4955us) service_pool.cc:227] Handling call
0807 11:11:14.112036 (+ 9696us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:14.112458 (+ 422us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:14.112503 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:14.113194 (+ 691us) log.cc:842] Serialized 26 byte log entry
0807 11:11:14.311809 (+198615us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:14.311852 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:14.311931 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:15.783949 (+1472018us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:11:15.807605 (+ 23656us) raft_consensus.cc:1671] finished
0807 11:11:15.807638 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:15.807852 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1447,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":114,"spinlock_wait_cycles":31677696,"thread_start_us":198217,"threads_started":1,"wal-append.queue_time_us":408005}
I20260807 11:11:15.810967 25808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53312 (request call id 1912) took 1042 ms. Trace:
I20260807 11:11:15.811354 25808 rpcz_store.cc:276] 0807 11:11:14.768075 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:14.773488 (+ 5413us) service_pool.cc:227] Handling call
0807 11:11:15.810773 (+1037285us) spinlock_profiling.cc:243] Waited 290 ms on lock 0x7b54000479d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:15.810908 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":667906432}
W20260807 11:11:15.804600 25806 consensus_meta.cc:287] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.629s user 0.008s sys 0.000s
I20260807 11:11:15.813488 25806 raft_consensus.cc:2454] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 7.
I20260807 11:11:15.813745 25931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1849) took 1535 ms. Trace:
I20260807 11:11:15.813943 30778 raft_consensus.cc:2741] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260807 11:11:15.814150 25931 rpcz_store.cc:276] 0807 11:11:14.278759 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:14.300031 (+ 21272us) service_pool.cc:227] Handling call
0807 11:11:14.313828 (+ 13797us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:14.597121 (+283293us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:15.813533 (+1216412us) spinlock_profiling.cc:243] Waited 191 ms on lock 0x7b5400408154. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:15.813689 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":440052608}
I20260807 11:11:15.564888 29915 consensus_queue.cc:237] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:15.807504 26033 rpcz_store.cc:276] 0807 11:11:14.122026 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:15.137547 (+1015521us) service_pool.cc:227] Handling call
0807 11:11:15.155730 (+ 18183us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:15.155937 (+ 207us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:15.155958 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:15.158255 (+ 2297us) log.cc:842] Serialized 26 byte log entry
0807 11:11:15.688381 (+530126us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:15.688437 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:15.688539 (+ 102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:15.806896 (+118357us) raft_consensus.cc:1671] finished
0807 11:11:15.806915 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:15.807066 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":460245,"threads_started":1,"wal-append.queue_time_us":612982}
I20260807 11:11:15.797394 25916 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 394) took 1197 ms. Trace:
I20260807 11:11:15.808043 25796 rpcz_store.cc:276] 0807 11:11:14.114878 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:14.501030 (+386152us) service_pool.cc:227] Handling call
0807 11:11:14.553650 (+ 52620us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:14.773459 (+219809us) raft_consensus.cc:1520] Early marking committed up to index 6
0807 11:11:14.825391 (+ 51932us) log.cc:842] Serialized 14 byte log entry
0807 11:11:15.347496 (+522105us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:15.409314 (+ 61818us) log.cc:842] Serialized 26 byte log entry
0807 11:11:15.409464 (+ 150us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:15.409495 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:15.409557 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:15.807471 (+397914us) raft_consensus.cc:1671] finished
0807 11:11:15.807489 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:15.807606 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28446,"prepare.queue_time_us":64239,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":50,"raft.queue_time_us":251398,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":5928,"thread_start_us":596710,"threads_started":2,"wal-append.queue_time_us":545314}
I20260807 11:11:15.822368 25916 rpcz_store.cc:276] 0807 11:11:14.599995 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:14.727583 (+127588us) service_pool.cc:227] Handling call
0807 11:11:14.727902 (+ 319us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:14.762241 (+ 34339us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:14.762298 (+ 57us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:14.762851 (+ 553us) log.cc:842] Serialized 26 byte log entry
0807 11:11:14.960267 (+197416us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:14.960306 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:14.960382 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:15.797106 (+836724us) raft_consensus.cc:1671] finished
0807 11:11:15.797136 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:15.797300 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":196509,"threads_started":1,"wal-append.queue_time_us":410852}
I20260807 11:11:15.824594 25916 raft_consensus.cc:1226] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 10 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 9. Current term is 10. Ops: []
I20260807 11:11:15.769662 30171 raft_consensus.cc:729] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:15.849604 30585 raft_consensus.cc:3032] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 LEADER]: Stepping down as leader of term 9
I20260807 11:11:15.850198 30585 raft_consensus.cc:729] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:15.849957 30171 consensus_queue.cc:260] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:15.851387 30171 raft_consensus.cc:3037] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:15.851271 30585 consensus_queue.cc:260] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:15.853068 30717 raft_consensus.cc:3032] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 11:11:15.853585 30717 raft_consensus.cc:729] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:15.854079 30780 leader_election.cc:288] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:15.855376 30770 leader_election.cc:288] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:15.828660 30756 consensus_queue.cc:1035] T cb81660c5a8b4365a928d2ef1e4d0be5 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260807 11:11:15.606361 30852 raft_consensus.cc:504] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:14.989881 26035 rpcz_store.cc:276] 0807 11:11:12.204218 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:13.762978 (+1558760us) service_pool.cc:227] Handling call
0807 11:11:13.763257 (+ 279us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:13.763465 (+ 208us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 11:11:13.864377 (+100912us) log.cc:842] Serialized 14 byte log entry
0807 11:11:14.753748 (+889371us) spinlock_profiling.cc:243] Waited 1.46 ms on lock 0x7b18002d3460. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbc77e345 00007f3fc55b5a39 00007f3fbc79b8bb 00007f3fbc799cbf 00007f3fbc79be22 00007f3fbf3455bb 00007f3fbf34b063 00007f3fbf34b57b 00007f3fbf6ce88a
0807 11:11:14.962239 (+208491us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:14.962290 (+ 51us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:14.962348 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:14.962421 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:14.962542 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38346,"prepare.queue_time_us":124594,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":790,"raft.run_wall_time_us":13519,"spinlock_wait_cycles":3350272,"thread_start_us":208269,"threads_started":1}
W20260807 11:11:15.828397 30666 consensus_meta.cc:287] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.769s user 0.011s sys 0.001s
I20260807 11:11:15.616122 30390 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:15.830310 25916 raft_consensus.cc:1226] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 10 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260807 11:11:15.830842 30278 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:15.832108 30856 consensus_queue.cc:1046] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:11:15.796195 30496 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:15.835862 25752 leader_election.cc:302] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:15.844000 30681 raft_consensus.cc:3032] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 LEADER]: Stepping down as leader of term 8
I20260807 11:11:15.859046 26035 raft_consensus.cc:1226] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7. Ops: []
W20260807 11:11:15.913194 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 77399.1us
I20260807 11:11:15.852644 30585 raft_consensus.cc:3037] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 FOLLOWER]: Advancing to term 10
I20260807 11:11:15.925556 26936 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
W20260807 11:11:15.844652 29851 consensus_meta.cc:287] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.779s user 0.007s sys 0.003s
W20260807 11:11:15.944011 25932 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1836) took 2074 ms (client timeout 2222 ms). Trace:
W20260807 11:11:15.944499 25932 rpcz_store.cc:269] 0807 11:11:13.869332 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:13.869463 (+ 131us) service_pool.cc:227] Handling call
0807 11:11:15.943800 (+2074337us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54001a2054. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:15.943937 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2490985856}
I20260807 11:11:15.945185 25918 raft_consensus.cc:1226] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:15.947634 30582 raft_consensus.cc:2726] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 11:11:15.950702 29417 consensus_queue.cc:1046] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260807 11:11:15.954284 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1898) took 1988 ms. Trace:
I20260807 11:11:15.954769 26039 rpcz_store.cc:276] 0807 11:11:13.965476 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:14.961917 (+996441us) service_pool.cc:227] Handling call
0807 11:11:14.997886 (+ 35969us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:15.036736 (+ 38850us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 11:11:15.068957 (+ 32221us) log.cc:842] Serialized 14 byte log entry
0807 11:11:15.143631 (+ 74674us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:15.143686 (+ 55us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:15.143761 (+ 75us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:15.837612 (+693851us) spinlock_profiling.cc:243] Waited 18.3 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:11:15.953089 (+115477us) spinlock_profiling.cc:243] Waited 9.19 ms on lock 0x7b1800268840. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbc769f96 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9
0807 11:11:15.954026 (+ 937us) spinlock_profiling.cc:243] Waited 38.5 ms on lock 0x7b540038ed54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:15.954060 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:15.954217 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":32031,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":82,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":9843,"spinlock_wait_cycles":151801472,"thread_start_us":74330,"threads_started":1,"wal-append.queue_time_us":420167}
I20260807 11:11:15.957901 30852 leader_election.cc:288] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:11:15.961236 25869 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.004s sys 0.012s
W20260807 11:11:15.961565 25869 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.518s user 0.004s sys 0.012s
W20260807 11:11:15.961884 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 528966us
I20260807 11:11:15.976199 25869 leader_election.cc:302] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
W20260807 11:11:15.976699 25801 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.556s user 0.004s sys 0.013s
W20260807 11:11:15.977165 25801 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.557s user 0.004s sys 0.013s
W20260807 11:11:15.981371 25920 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.669s user 0.000s sys 0.017s
I20260807 11:11:15.981812 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:15.983385 25806 raft_consensus.cc:2396] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 11:11:15.926256 30770 raft_consensus.cc:2781] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 11:11:15.925822 30693 raft_consensus.cc:3037] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Advancing to term 8
W20260807 11:11:15.984436 25920 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.672s user 0.000s sys 0.017s
I20260807 11:11:15.918372 30681 raft_consensus.cc:729] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:15.988837 26036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1981) took 2026 ms. Trace:
I20260807 11:11:15.989295 30681 consensus_queue.cc:260] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:15.763300 30842 raft_consensus.cc:504] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:15.787427 30743 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:15.989235 26036 rpcz_store.cc:276] 0807 11:11:13.962055 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:14.948442 (+986387us) service_pool.cc:227] Handling call
0807 11:11:14.948911 (+ 469us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:14.950852 (+ 1941us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:14.950899 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:14.951545 (+ 646us) log.cc:842] Serialized 26 byte log entry
0807 11:11:15.111165 (+159620us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:15.111320 (+ 155us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:15.111610 (+ 290us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:15.988570 (+876960us) raft_consensus.cc:1671] finished
0807 11:11:15.988593 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:15.988755 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1412,"raft.queue_time_us":3220,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":113,"thread_start_us":158752,"threads_started":1,"wal-append.queue_time_us":296289}
I20260807 11:11:15.929358 30858 raft_consensus.cc:3037] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:15.991143 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:15.935060 30697 raft_consensus.cc:2726] T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:15.915220 30854 raft_consensus.cc:2726] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:11:16.014370 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:16.026373 26036 raft_consensus.cc:2454] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 4.
I20260807 11:11:15.960513 30766 raft_consensus.cc:3032] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Stepping down as leader of term 6
I20260807 11:11:16.019150 26039 raft_consensus.cc:2454] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
W20260807 11:11:15.990657 30556 consensus_meta.cc:287] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 1.033s user 0.009s sys 0.005s
I20260807 11:11:16.027869 25752 leader_election.cc:302] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:15.987087 30852 raft_consensus.cc:2781] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader election won for term 7
I20260807 11:11:15.924744 30496 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:15.853291 30724 consensus_queue.cc:237] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:16.060204 25981 leader_election.cc:398] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195) with higher term. Message: Invalid argument: T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 5. Current term is 6.
W20260807 11:11:16.049880 30726 consensus_meta.cc:287] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.838s user 0.007s sys 0.008s
I20260807 11:11:16.015693 30415 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:16.042541 25752 leader_election.cc:302] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:15.964584 29851 raft_consensus.cc:504] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:16.074685 25806 raft_consensus.cc:1226] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:16.014099 30770 raft_consensus.cc:686] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 7 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:16.041716 26036 raft_consensus.cc:1226] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 7. Current term is 8. Ops: []
I20260807 11:11:16.062211 25808 raft_consensus.cc:1226] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260807 11:11:16.035596 30766 raft_consensus.cc:729] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:16.072613 25921 raft_consensus.cc:3032] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 7 LEADER]: Stepping down as leader of term 7
W20260807 11:11:16.074820 25750 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.653s user 0.006s sys 0.022s
I20260807 11:11:16.080333 25921 raft_consensus.cc:729] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
W20260807 11:11:16.080533 25750 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.658s user 0.006s sys 0.023s
W20260807 11:11:16.080920 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 658957us
I20260807 11:11:16.072491 25796 raft_consensus.cc:1261] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260807 11:11:16.055305 29857 raft_consensus.cc:3032] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 11:11:16.045646 30856 raft_consensus.cc:2781] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260807 11:11:16.069767 26938 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.770s user 0.007s sys 0.019s
W20260807 11:11:16.087829 26938 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.788s user 0.007s sys 0.019s
W20260807 11:11:16.089284 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 828650us
I20260807 11:11:16.082671 25750 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:16.093276 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:16.081375 25921 consensus_queue.cc:260] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:16.093441 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:15.981705 30717 consensus_queue.cc:260] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:16.078887 30756 raft_consensus.cc:2781] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260807 11:11:16.095757 30794 consensus_meta.cc:287] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 1.029s user 0.020s sys 0.000s
I20260807 11:11:16.083390 29857 raft_consensus.cc:729] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
W20260807 11:11:16.149843 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 60829.8us
I20260807 11:11:16.084298 30256 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:16.087661 30856 raft_consensus.cc:482] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:16.156651 26042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 549) took 2953 ms. Trace:
I20260807 11:11:16.121347 25921 raft_consensus.cc:3037] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:16.121582 25916 raft_consensus.cc:2379] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 9: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:11:16.122278 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
W20260807 11:11:16.105046 25798 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.800s user 0.003s sys 0.018s
I20260807 11:11:16.157807 25801 raft_consensus.cc:3037] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
W20260807 11:11:16.158104 25798 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.861s user 0.004s sys 0.019s
I20260807 11:11:16.119481 25981 leader_election.cc:398] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195) with higher term. Message: Invalid argument: T adab2949574e4c6badfcf68e51b9c072 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 6. Current term is 7.
I20260807 11:11:16.154328 25806 raft_consensus.cc:1226] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 5. Current term is 6. Ops: []
I20260807 11:11:16.115607 30400 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:16.088037 30415 consensus_queue.cc:1035] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20260807 11:11:16.161432 25981 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:11:16.139446 30756 raft_consensus.cc:482] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
W20260807 11:11:16.171516 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 88940.2us
I20260807 11:11:16.182087 25750 leader_election.cc:302] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:16.170014 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:16.153945 30717 raft_consensus.cc:3037] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:16.161377 30856 raft_consensus.cc:3037] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:11:16.076582 30726 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:16.187399 25750 leader_election.cc:302] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:16.174616 25801 raft_consensus.cc:2454] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 7.
W20260807 11:11:16.175374 30320 consensus_meta.cc:287] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.998s user 0.012s sys 0.005s
I20260807 11:11:16.184659 25981 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:11:16.259861 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 75301.2us
W20260807 11:11:16.153168 30154 consensus_queue.cc:1159] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b22d69db45194908ad3d345eba9badd9" responder_term: 7 status { last_received { term: 6 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260807 11:11:16.173537 30390 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:16.249042 25921 raft_consensus.cc:1261] T 54a291aa09434b06adbb86586dacec0e P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260807 11:11:16.180035 30847 raft_consensus.cc:2726] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 11:11:16.185127 30756 raft_consensus.cc:3037] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:16.178133 30708 raft_consensus.cc:482] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:16.225819 30562 raft_consensus.cc:2726] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:16.228684 30776 raft_consensus.cc:2741] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260807 11:11:16.290429 25925 raft_consensus.cc:2354] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 8. Current term is 9.
I20260807 11:11:16.151042 30794 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:16.284127 30320 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:16.294360 30562 raft_consensus.cc:2781] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Leader election won for term 9
I20260807 11:11:16.294762 30708 raft_consensus.cc:504] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:16.283932 30475 raft_consensus.cc:2726] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:16.172187 30766 consensus_queue.cc:260] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:16.239392 30556 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:16.322253 25981 leader_election.cc:302] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
W20260807 11:11:16.289609 30764 log.cc:924] Time spent T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 1.298s user 0.001s sys 0.003s
I20260807 11:11:16.174566 30400 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:16.345208 25803 raft_consensus.cc:1226] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 6. Current term is 7. Ops: []
W20260807 11:11:16.348697 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 46273.9us
I20260807 11:11:16.350225 25981 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:16.351723 25981 leader_election.cc:302] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:11:16.343557 28964 consensus_meta.cc:287] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 1.099s user 0.004s sys 0.008s
I20260807 11:11:16.189464 30770 consensus_queue.cc:237] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:16.225813 29857 consensus_queue.cc:260] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:16.341027 29851 leader_election.cc:288] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:16.344698 30776 raft_consensus.cc:2726] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:16.340716 30562 raft_consensus.cc:686] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [term 9 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:16.353039 30766 raft_consensus.cc:3037] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
W20260807 11:11:16.404244 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 216832us
W20260807 11:11:16.417405 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 593941us
W20260807 11:11:16.422982 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 71322us
I20260807 11:11:16.354933 30475 raft_consensus.cc:2781] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:16.346652 30842 leader_election.cc:288] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:16.446203 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:16.363283 28964 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:16.431433 25916 raft_consensus.cc:1226] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5. Ops: [3.2-4.3]
I20260807 11:11:16.431396 30776 raft_consensus.cc:2726] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260807 11:11:16.398669 30884 consensus_queue.cc:1159] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b22d69db45194908ad3d345eba9badd9" responder_term: 8 status { last_received { term: 7 index: 4 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 7. Current term is 8. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260807 11:11:16.443169 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:16.452491 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:16.475958 30475 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:16.458006 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
W20260807 11:11:16.490429 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 62895.7us
W20260807 11:11:16.480875 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 303826us
W20260807 11:11:16.510558 25981 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.120.130:36465 timed out after 2.104s (SENT) [suppressed 22 similar messages]
I20260807 11:11:16.521463 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
W20260807 11:11:16.400527 30773 consensus_meta.cc:287] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.895s user 0.010s sys 0.001s
I20260807 11:11:16.525921 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:16.526242 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:16.346531 30556 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:16.537631 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:16.526620 30847 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:16.538975 25797 raft_consensus.cc:1226] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 7. Current term is 8. Ops: []
I20260807 11:11:16.408541 29417 consensus_queue.cc:1035] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.134s
I20260807 11:11:16.572503 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:16.741940 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6408e9acee46adacd2bbd81669f27a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:16.743069 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:16.579968 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1964) took 4160 ms. Trace:
I20260807 11:11:16.568107 25750 leader_election.cc:302] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:16.686941 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a74fdae5d44c8a9b375e1cfdea1e9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:16.747251 25916 raft_consensus.cc:2396] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260807 11:11:16.660813 26035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1988) took 1403 ms. Trace:
I20260807 11:11:16.573684 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:16.747861 30320 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:16.659175 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 563) took 1681 ms. Trace:
I20260807 11:11:16.721338 25921 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 10.
I20260807 11:11:16.686347 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:16.744335 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:16.751500 25800 raft_consensus.cc:2379] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 7: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:11:16.665642 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 396) took 1406 ms. Trace:
I20260807 11:11:16.503666 30496 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:11:16.521077 30718 consensus_meta.cc:287] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 1.074s user 0.012s sys 0.000s
I20260807 11:11:16.748636 30922 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:16.746044 26034 rpcz_store.cc:276] 0807 11:11:12.419171 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:13.808878 (+1389707us) service_pool.cc:227] Handling call
0807 11:11:13.809237 (+ 359us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:13.859319 (+ 50082us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:13.859363 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:13.866271 (+ 6908us) log.cc:842] Serialized 26 byte log entry
0807 11:11:14.088118 (+221847us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:14.088174 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:16.550206 (+2462032us) spinlock_profiling.cc:243] Waited 49.6 ms on lock 0x7b540022a454. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:16.572270 (+ 22064us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:16.572338 (+ 68us) raft_consensus.cc:1671] finished
0807 11:11:16.572356 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:16.579879 (+ 7523us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1123,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":10538,"spinlock_wait_cycles":114193664,"thread_start_us":187860,"threads_started":1,"wal-append.queue_time_us":420941,"wal-append.run_cpu_time_us":2198,"wal-append.run_wall_time_us":1093742}
I20260807 11:11:16.754272 26037 rpcz_store.cc:276] 0807 11:11:14.978169 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:15.811293 (+833124us) service_pool.cc:227] Handling call
0807 11:11:15.811608 (+ 315us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:15.812002 (+ 394us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:15.812030 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:15.812531 (+ 501us) log.cc:842] Serialized 26 byte log entry
0807 11:11:16.261149 (+448618us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:16.261377 (+ 228us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:16.261472 (+ 95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:16.658935 (+397463us) raft_consensus.cc:1671] finished
0807 11:11:16.658956 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:16.659086 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1048,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":140,"thread_start_us":447019,"threads_started":1,"wal-append.queue_time_us":844600}
I20260807 11:11:16.771339 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:16.772336 26037 raft_consensus.cc:2454] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 7.
I20260807 11:11:16.773633 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:11:16.638064 30885 log.cc:924] Time spent T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.336s user 0.001s sys 0.000s
I20260807 11:11:16.538959 30475 raft_consensus.cc:3037] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
W20260807 11:11:16.557430 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 214094us
W20260807 11:11:16.643846 30693 consensus_meta.cc:287] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.646s user 0.007s sys 0.000s
W20260807 11:11:16.766010 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 197995us
I20260807 11:11:16.777333 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:16.795642 25750 leader_election.cc:302] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:16.797627 25750 leader_election.cc:302] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters: eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:11:16.565368 30820 consensus_meta.cc:287] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 1.142s user 0.013s sys 0.004s
W20260807 11:11:16.641353 30883 log.cc:924] Time spent T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.402s user 0.001s sys 0.000s
I20260807 11:11:16.752045 26035 rpcz_store.cc:276] 0807 11:11:15.257404 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:15.923331 (+665927us) service_pool.cc:227] Handling call
0807 11:11:15.934644 (+ 11313us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:15.942754 (+ 8110us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:15.942785 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:15.943334 (+ 549us) log.cc:842] Serialized 26 byte log entry
0807 11:11:16.249289 (+305955us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:16.249330 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:16.249419 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:16.660558 (+411139us) raft_consensus.cc:1671] finished
0807 11:11:16.660583 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:16.660730 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1455,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":95,"thread_start_us":305592,"threads_started":1,"wal-append.queue_time_us":596745}
W20260807 11:11:16.599932 30863 log.cc:924] Time spent T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.458s user 0.001s sys 0.001s
W20260807 11:11:16.637643 30868 log.cc:924] Time spent T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.283s user 0.000s sys 0.001s
W20260807 11:11:16.638489 30887 log.cc:924] Time spent T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260807 11:11:16.761507 25917 rpcz_store.cc:276] 0807 11:11:15.261797 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:15.805124 (+543327us) service_pool.cc:227] Handling call
0807 11:11:15.805468 (+ 344us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:16.116081 (+310613us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:16.116130 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:16.116691 (+ 561us) log.cc:842] Serialized 26 byte log entry
0807 11:11:16.348950 (+232259us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:16.349505 (+ 555us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:16.350338 (+ 833us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:16.665384 (+315046us) raft_consensus.cc:1671] finished
0807 11:11:16.665417 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:16.665576 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12449,"raft.queue_time_us":124144,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":29075,"thread_start_us":477084,"threads_started":2,"wal-append.queue_time_us":513780}
W20260807 11:11:16.782747 30717 consensus_meta.cc:287] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.555s user 0.010s sys 0.000s
I20260807 11:11:16.635229 30562 consensus_queue.cc:237] T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:16.646848 30400 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:16.782549 30718 raft_consensus.cc:504] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:16.856228 30718 leader_election.cc:288] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:16.856266 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:16.857017 30718 raft_consensus.cc:2781] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 11:11:16.857339 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:16.857836 25801 raft_consensus.cc:1226] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 4. Current term is 5. Ops: []
I20260807 11:11:16.783231 30776 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:16.783418 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:16.667346 30726 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 9 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:16.801749 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:16.660372 30794 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:16.663640 30708 leader_election.cc:288] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:16.833470 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:16.769997 29851 raft_consensus.cc:2726] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 11:11:16.839172 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:16.763674 28964 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:16.883926 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31f65eddcdf48f6a64e0a1ddc0c6695" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:16.836598 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:16.827082 30415 raft_consensus.cc:2726] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 11:11:16.545396 30773 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:16.851511 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:16.852074 30881 raft_consensus.cc:482] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:16.853094 30717 consensus_queue.cc:1035] T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260807 11:11:16.853775 30856 consensus_meta.cc:287] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.614s user 0.009s sys 0.000s
I20260807 11:11:16.783377 30922 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:16.803777 30693 raft_consensus.cc:2726] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 11:11:16.872814 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:16.846032 26036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1992) took 1587 ms. Trace:
I20260807 11:11:16.552561 30847 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:16.810158 30820 raft_consensus.cc:504] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:16.891177 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:16.891925 25800 raft_consensus.cc:1226] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 7. Current term is 8. Ops: []
I20260807 11:11:16.942795 25932 raft_consensus.cc:1226] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 7. Current term is 8. Ops: []
I20260807 11:11:16.935988 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:16.877187 25808 raft_consensus.cc:2454] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 7.
I20260807 11:11:16.943660 26034 raft_consensus.cc:2454] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 8.
I20260807 11:11:16.937783 26987 raft_consensus.cc:2454] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 10.
I20260807 11:11:16.938035 26938 leader_election.cc:302] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 4fd6381544574ddca559ce63b9243efc
W20260807 11:11:16.940161 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 44964.1us
I20260807 11:11:16.853107 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:16.886005 30903 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:16.895354 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:16.941136 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:16.834357 30864 consensus_queue.cc:1046] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.061s
I20260807 11:11:16.897713 30415 raft_consensus.cc:482] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:16.957118 30847 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:11:16.958618 30908 log.cc:924] Time spent T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20260807 11:11:16.961181 30773 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:16.961488 30390 raft_consensus.cc:2781] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260807 11:11:16.962162 30905 log.cc:924] Time spent T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.213s user 0.001s sys 0.001s
I20260807 11:11:16.957173 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:16.963351 25806 raft_consensus.cc:2454] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 7.
I20260807 11:11:16.936206 26036 rpcz_store.cc:276] 0807 11:11:15.258609 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:16.079322 (+820713us) service_pool.cc:227] Handling call
0807 11:11:16.079622 (+ 300us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:16.087506 (+ 7884us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 11:11:16.391322 (+303816us) log.cc:842] Serialized 14 byte log entry
0807 11:11:16.845563 (+454241us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:16.845634 (+ 71us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:16.845698 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:16.845795 (+ 97us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:16.845956 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":482819,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":245739,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":129,"thread_start_us":692761,"threads_started":3,"wal-append.queue_time_us":239259}
I20260807 11:11:16.994637 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:16.995384 26036 raft_consensus.cc:3037] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:16.979072 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:16.899027 30556 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:16.981832 30937 raft_consensus.cc:482] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:16.982857 25786 raft_consensus.cc:1226] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5. Ops: []
I20260807 11:11:16.985286 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:17.010387 26034 raft_consensus.cc:3037] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:11:16.986537 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:17.011905 25917 raft_consensus.cc:2379] T f35a17e1683e41f4b146284f5cf09c58 P 4fd6381544574ddca559ce63b9243efc [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 10: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
W20260807 11:11:16.992327 30771 consensus_meta.cc:287] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 1.695s user 0.013s sys 0.005s
I20260807 11:11:16.892417 30708 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:16.894595 29851 raft_consensus.cc:482] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:11:17.002434 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 491822us
I20260807 11:11:16.986711 25750 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:16.872268 30776 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:16.983412 30820 leader_election.cc:288] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:17.042785 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:17.046996 26036 raft_consensus.cc:2454] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 7.
I20260807 11:11:16.993136 30390 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:16.983286 30415 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:17.045418 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 58822us
I20260807 11:11:17.048552 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:17.049257 26036 raft_consensus.cc:3037] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Advancing to term 8
W20260807 11:11:17.030648 30766 consensus_meta.cc:287] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.597s user 0.009s sys 0.001s
I20260807 11:11:17.050668 29851 raft_consensus.cc:504] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:17.053385 30898 log.cc:924] Time spent T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.111s user 0.000s sys 0.000s
I20260807 11:11:17.053939 30776 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:17.054525 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1990) took 1796 ms. Trace:
W20260807 11:11:17.059018 30756 consensus_meta.cc:287] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.763s user 0.010s sys 0.000s
W20260807 11:11:17.060681 30585 consensus_meta.cc:287] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 1.098s user 0.007s sys 0.005s
I20260807 11:11:17.065284 30922 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:16.971331 30795 consensus_queue.cc:1035] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.060s
I20260807 11:11:16.974939 30903 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:16.897339 30856 raft_consensus.cc:504] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:17.075682 26034 raft_consensus.cc:2454] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:11:17.051426 25750 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:17.108750 25916 raft_consensus.cc:2454] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 7.
I20260807 11:11:17.041952 30794 raft_consensus.cc:482] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:17.111799 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:17.113054 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:11:17.115839 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 65410.7us
I20260807 11:11:17.050958 28964 raft_consensus.cc:2726] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 11:11:17.116487 26034 raft_consensus.cc:2454] T 6eed5249afac4e16a4b17938ad6e1353 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:11:17.107673 26039 rpcz_store.cc:276] 0807 11:11:15.258017 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:16.039909 (+781892us) service_pool.cc:227] Handling call
0807 11:11:16.061993 (+ 22084us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:16.062360 (+ 367us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:16.062391 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:16.073703 (+ 11312us) log.cc:842] Serialized 26 byte log entry
0807 11:11:16.401690 (+327987us) raft_consensus.cc:1631] Marking committed up to 8
0807 11:11:16.401746 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:16.401827 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:17.054303 (+652476us) raft_consensus.cc:1671] finished
0807 11:11:17.054325 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:17.054460 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":113,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":7592,"thread_start_us":249996,"threads_started":1,"wal-append.queue_time_us":656405}
I20260807 11:11:16.898890 30881 raft_consensus.cc:504] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:17.119125 30940 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Advancing to term 10
I20260807 11:11:17.121500 25917 raft_consensus.cc:2354] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7.
I20260807 11:11:17.070814 30390 raft_consensus.cc:3037] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Advancing to term 9
W20260807 11:11:17.125458 25750 leader_election.cc:334] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Timed out: RequestConsensusVote RPC to 127.23.120.130:36465 timed out after 1.730s (SENT)
I20260807 11:11:17.124260 25752 leader_election.cc:302] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc
I20260807 11:11:17.130993 30903 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:17.008364 30937 raft_consensus.cc:504] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:17.132289 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:17.134289 25917 raft_consensus.cc:2454] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 7.
I20260807 11:11:17.135601 25752 leader_election.cc:302] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:17.136844 25752 leader_election.cc:302] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:17.139468 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:17.140293 25917 raft_consensus.cc:2362] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6. Re-sending same reply.
I20260807 11:11:17.144723 28964 raft_consensus.cc:2781] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 11:11:17.160066 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1911) took 1854 ms. Trace:
I20260807 11:11:17.051251 30708 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:17.143370 26034 raft_consensus.cc:1226] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 4. Ops: []
I20260807 11:11:17.142443 26036 raft_consensus.cc:2454] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 8.
I20260807 11:11:17.175137 29851 leader_election.cc:288] T 93d531084c9447b78cffb286d6b966b7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:17.113044 30756 raft_consensus.cc:504] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:17.164938 25750 leader_election.cc:302] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:17.190186 25918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 391) took 2601 ms. Trace:
I20260807 11:11:17.173799 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:17.176419 26039 rpcz_store.cc:276] 0807 11:11:15.305761 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:17.124248 (+1818487us) service_pool.cc:227] Handling call
0807 11:11:17.157378 (+ 33130us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:17.159655 (+ 2277us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:17.159689 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:17.159744 (+ 55us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:11:17.159787 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:17.159858 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:17.159992 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:17.190670 25918 rpcz_store.cc:276] 0807 11:11:14.589069 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:14.719096 (+130027us) service_pool.cc:227] Handling call
0807 11:11:15.945002 (+1225906us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:15.945612 (+ 610us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:17.189965 (+1244353us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54001a2050. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:17.190121 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2818452864}
I20260807 11:11:17.139070 30838 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:17.172874 30475 raft_consensus.cc:504] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:17.184710 26980 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 10.
I20260807 11:11:17.186439 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:17.123431 30922 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:17.207849 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:17.142949 30717 raft_consensus.cc:2781] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election won for term 5
W20260807 11:11:17.163548 30858 consensus_meta.cc:287] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 1.123s user 0.005s sys 0.007s
I20260807 11:11:17.203723 26036 raft_consensus.cc:2454] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:11:17.190410 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1909) took 1916 ms. Trace:
I20260807 11:11:17.193101 25931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 397) took 1930 ms. Trace:
I20260807 11:11:17.195478 30946 raft_consensus.cc:2781] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:17.197162 30708 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:17.198002 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:17.211560 25931 rpcz_store.cc:276] 0807 11:11:15.262792 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:15.815185 (+552393us) service_pool.cc:227] Handling call
0807 11:11:15.829999 (+ 14814us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:16.037347 (+207348us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:16.037391 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:16.046941 (+ 9550us) log.cc:842] Serialized 26 byte log entry
0807 11:11:16.364705 (+317764us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:16.364871 (+ 166us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:17.067523 (+702652us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b5400053cd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:17.067578 (+ 55us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:17.184310 (+116732us) raft_consensus.cc:1671] finished
0807 11:11:17.184329 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:17.193015 (+ 8686us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":476098432,"thread_start_us":267182,"threads_started":1,"wal-append.queue_time_us":780740}
I20260807 11:11:17.216055 25800 raft_consensus.cc:2454] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 9.
W20260807 11:11:17.198717 25925 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39742 (request call id 1864) took 2156 ms (client timeout 1730 ms). Trace:
W20260807 11:11:17.208833 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 44139.7us
I20260807 11:11:17.143282 30937 leader_election.cc:288] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:17.208793 25752 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:17.229604 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:17.230820 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:17.228176 30776 raft_consensus.cc:482] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:17.214035 30795 raft_consensus.cc:2726] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 11:11:17.236479 30776 raft_consensus.cc:504] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:17.238139 30742 log.cc:924] Time spent T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 2.498s user 0.010s sys 0.004s
I20260807 11:11:17.238831 30776 leader_election.cc:288] T e0766ccf18b84e41ac68eb6599a834c5 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:17.235634 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:17.243080 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:17.244068 30856 leader_election.cc:288] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:17.244553 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:17.221156 30756 leader_election.cc:288] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:17.215711 30946 raft_consensus.cc:482] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:17.123916 30794 raft_consensus.cc:504] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:17.223011 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:17.269766 25918 raft_consensus.cc:2379] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 7: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
W20260807 11:11:17.216960 25925 rpcz_store.cc:269] 0807 11:11:15.041909 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:15.049742 (+ 7833us) service_pool.cc:227] Handling call
0807 11:11:17.196738 (+2146996us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54003b1ad4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:17.198631 (+ 1893us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2708264832}
I20260807 11:11:17.269894 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:17.271060 25917 raft_consensus.cc:2454] T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 5.
I20260807 11:11:17.233577 30475 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:17.209230 30838 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:17.273365 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:17.274178 25925 raft_consensus.cc:2379] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 5: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:11:17.275146 30838 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:17.247733 25797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1924) took 1898 ms. Trace:
I20260807 11:11:17.215129 30922 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:17.275801 25797 rpcz_store.cc:276] 0807 11:11:15.349666 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:15.349826 (+ 160us) service_pool.cc:227] Handling call
0807 11:11:15.403423 (+ 53597us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:16.739020 (+1335597us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:17.247500 (+508480us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54001ff754. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:17.247675 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2601652096}
I20260807 11:11:17.277503 30922 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:17.279213 30630 consensus_queue.cc:260] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:11:17.296232 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 78954.3us
I20260807 11:11:17.290679 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:17.210992 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1912) took 1905 ms. Trace:
I20260807 11:11:17.251080 30881 leader_election.cc:288] T 66d10b6b12404f0488b9f83c8080069f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:11:17.254232 30889 log.cc:924] Time spent T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.822s user 0.001s sys 0.000s
I20260807 11:11:17.244438 30743 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 9 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:17.256067 26979 raft_consensus.cc:1226] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 LEADER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 7. Current term is 8. Ops: []
I20260807 11:11:17.213223 26034 rpcz_store.cc:276] 0807 11:11:15.273558 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:17.126835 (+1853277us) service_pool.cc:227] Handling call
0807 11:11:17.143145 (+ 16310us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:17.190165 (+ 47020us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:17.190350 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:17.223904 30858 raft_consensus.cc:2726] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:17.258334 30721 raft_consensus.cc:3032] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Stepping down as leader of term 6
I20260807 11:11:17.263921 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 559) took 2681 ms. Trace:
I20260807 11:11:17.265185 30940 raft_consensus.cc:2726] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 11:11:17.224450 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:17.272545 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:17.325927 25796 raft_consensus.cc:1261] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 11:11:17.275259 25925 raft_consensus.cc:1226] T 409fcb387861467396e84e2b22b8f9be P 4fd6381544574ddca559ce63b9243efc [term 8 LEADER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 7. Current term is 8. Ops: []
I20260807 11:11:17.245800 30415 leader_election.cc:288] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:17.311800 26039 rpcz_store.cc:276] 0807 11:11:15.306051 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:17.192283 (+1886232us) service_pool.cc:227] Handling call
0807 11:11:17.210411 (+ 18128us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:17.210598 (+ 187us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:17.210631 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:17.210682 (+ 51us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:11:17.210746 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:17.210810 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:17.210924 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:11:17.313823 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 105228us
I20260807 11:11:17.329886 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:11:17.323042 25981 connection.cc:232] RPC call timeout handler was delayed by -4.0804855148307979s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1522s for a response
I20260807 11:11:17.332155 25917 raft_consensus.cc:1226] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 10 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
W20260807 11:11:17.332442 25981 connection.cc:232] RPC call timeout handler was delayed by -4.0651855147443712s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1539s for a response
I20260807 11:11:17.324729 26038 rpcz_store.cc:276] 0807 11:11:14.582372 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:15.503908 (+921536us) service_pool.cc:227] Handling call
0807 11:11:15.565955 (+ 62047us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:15.566391 (+ 436us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:15.566424 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:15.611875 (+ 45451us) log.cc:842] Serialized 26 byte log entry
0807 11:11:16.059208 (+447333us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:16.059262 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:16.059360 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:16.864006 (+804646us) raft_consensus.cc:1671] finished
0807 11:11:16.864044 (+ 38us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:17.263613 (+399569us) spinlock_profiling.cc:243] Waited 61.7 ms on lock 0x7b5400312250. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c3a09 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:17.263847 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":114,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":26517,"spinlock_wait_cycles":141908736,"thread_start_us":294516,"threads_started":1,"wal-append.queue_time_us":594873}
W20260807 11:11:17.333366 25981 connection.cc:232] RPC call timeout handler was delayed by -3.9949855147860944s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1617s for a response
W20260807 11:11:17.333879 25981 connection.cc:232] RPC call timeout handler was delayed by -3.8419855148531497s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1787s for a response
W20260807 11:11:17.334290 25981 connection.cc:232] RPC call timeout handler was delayed by -3.7141855147201568s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19290000000000002s for a response
W20260807 11:11:17.334794 25981 connection.cc:232] RPC call timeout handler was delayed by -3.4693855147343129s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22010000000000002s for a response
W20260807 11:11:17.335197 25981 connection.cc:232] RPC call timeout handler was delayed by -3.450485514709726s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2222s for a response
I20260807 11:11:17.336272 25981 leader_election.cc:302] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:11:17.340653 25981 leader_election.cc:302] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:17.285635 30398 raft_consensus.cc:3037] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 FOLLOWER]: Advancing to term 10
I20260807 11:11:17.256999 28964 raft_consensus.cc:686] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [term 7 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:17.287889 29857 raft_consensus.cc:3037] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:17.280709 30630 raft_consensus.cc:3037] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 FOLLOWER]: Advancing to term 10
I20260807 11:11:17.351776 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:17.352190 25918 raft_consensus.cc:1226] T cbfca1b0bf4c479a834433a4c0ed66ac P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 8. Current term is 9. Ops: [6.5-8.6]
I20260807 11:11:17.352451 25981 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:17.352744 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:17.357992 26034 raft_consensus.cc:2396] T 93d531084c9447b78cffb286d6b966b7 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260807 11:11:17.358366 25981 leader_election.cc:302] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:11:17.358841 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:17.359499 26034 raft_consensus.cc:3037] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:17.317596 30717 raft_consensus.cc:686] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:17.328855 30475 raft_consensus.cc:482] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:17.355346 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:17.314232 30946 raft_consensus.cc:3037] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:17.352891 25931 raft_consensus.cc:1226] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 4. Current term is 5. Ops: []
I20260807 11:11:17.365059 25752 leader_election.cc:302] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:17.324038 30881 raft_consensus.cc:2781] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 11:11:17.367967 25801 raft_consensus.cc:1261] T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260807 11:11:17.377130 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:17.321561 30922 raft_consensus.cc:482] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:17.375157 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:17.378129 26038 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:11:17.324579 30721 raft_consensus.cc:729] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:17.257207 30390 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:17.378782 25917 raft_consensus.cc:1226] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260807 11:11:17.360478 30937 raft_consensus.cc:2726] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 11:11:17.380477 30927 raft_consensus.cc:2726] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:17.382068 26035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1908) took 2110 ms. Trace:
I20260807 11:11:17.395402 28964 consensus_queue.cc:237] T b8330847089e41b0811e28dc30529234 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:17.384804 26039 raft_consensus.cc:2379] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 7: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:11:17.313853 30718 raft_consensus.cc:686] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [term 7 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:17.356563 30776 raft_consensus.cc:2781] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:17.378772 25750 leader_election.cc:398] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465) with higher term. Message: Invalid argument: T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7.
I20260807 11:11:17.400389 26036 raft_consensus.cc:2454] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:11:17.405818 25750 leader_election.cc:401] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260807 11:11:17.392402 30708 raft_consensus.cc:2781] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 11:11:17.400308 26035 rpcz_store.cc:276] 0807 11:11:15.271980 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:16.803543 (+1531563us) service_pool.cc:227] Handling call
0807 11:11:16.839747 (+ 36204us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:16.839992 (+ 245us) raft_consensus.cc:1520] Early marking committed up to index 5
0807 11:11:16.884030 (+ 44038us) log.cc:842] Serialized 14 byte log entry
0807 11:11:17.381645 (+497615us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:17.381724 (+ 79us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:11:17.381790 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:17.381872 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:17.382001 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":32191,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":120043,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":16581,"spinlock_wait_cycles":1002752,"thread_start_us":489701,"threads_started":2,"wal-append.queue_time_us":310196}
I20260807 11:11:17.377319 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:11:17.410663 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 52392.3us
I20260807 11:11:17.411594 25981 leader_election.cc:302] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:17.421420 26035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1998) took 1332 ms. Trace:
I20260807 11:11:17.422118 26035 rpcz_store.cc:276] 0807 11:11:16.089074 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:17.406777 (+1317703us) service_pool.cc:227] Handling call
0807 11:11:17.420827 (+ 14050us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:17.421031 (+ 204us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:17.421066 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:17.421121 (+ 55us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:17.421172 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:17.421234 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:17.421352 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:17.378417 30475 raft_consensus.cc:504] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:17.436825 26936 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:17.404884 30934 raft_consensus.cc:2781] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 11:11:17.408901 30708 raft_consensus.cc:482] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:17.461377 26035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 567) took 1369 ms. Trace:
I20260807 11:11:17.434237 25981 leader_election.cc:302] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:17.405637 30776 raft_consensus.cc:482] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:17.461854 26035 rpcz_store.cc:276] 0807 11:11:16.090639 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:17.424528 (+1333889us) service_pool.cc:227] Handling call
0807 11:11:17.459253 (+ 34725us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:17.459434 (+ 181us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:17.459459 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:17.459502 (+ 43us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:17.459538 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:17.459599 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:17.459721 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:17.413025 30881 raft_consensus.cc:686] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [term 8 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:17.421202 30927 raft_consensus.cc:2726] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:11:17.459527 25750 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:17.435346 30795 raft_consensus.cc:3037] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:17.448799 26034 raft_consensus.cc:2454] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 7.
I20260807 11:11:17.465003 25801 raft_consensus.cc:1226] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260807 11:11:17.457453 30937 raft_consensus.cc:482] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:17.388806 30922 raft_consensus.cc:504] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:17.465451 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:17.465507 30708 raft_consensus.cc:3037] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:17.467584 30776 raft_consensus.cc:3037] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:17.478531 26034 raft_consensus.cc:1261] T 54a291aa09434b06adbb86586dacec0e P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260807 11:11:17.467720 30171 raft_consensus.cc:2781] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 11:11:17.490334 30922 leader_election.cc:288] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:17.472421 30940 raft_consensus.cc:2726] T cf86a21774c34f33a6540d1dad0de0c4 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:17.490916 25803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1920) took 2675 ms. Trace:
I20260807 11:11:17.512303 25803 rpcz_store.cc:276] 0807 11:11:14.815673 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:15.046329 (+230656us) service_pool.cc:227] Handling call
0807 11:11:15.125008 (+ 78679us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:16.345744 (+1220736us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:17.490672 (+1144928us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54002d3cd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:17.490851 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2805607296}
I20260807 11:11:17.480798 30927 raft_consensus.cc:2781] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 10 FOLLOWER]: Leader election won for term 10
I20260807 11:11:17.492978 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:17.480381 25752 leader_election.cc:302] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:17.462311 30934 raft_consensus.cc:482] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:17.493829 25981 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:17.432621 30717 consensus_queue.cc:237] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:17.510907 30171 raft_consensus.cc:482] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:17.521883 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
W20260807 11:11:17.526211 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 45937us
I20260807 11:11:17.472168 30924 consensus_queue.cc:1046] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.051s
I20260807 11:11:17.531503 25797 raft_consensus.cc:3037] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:17.537967 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:17.521869 30390 leader_election.cc:288] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 9 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:17.471203 30718 consensus_queue.cc:237] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:11:17.543366 26037 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.655s user 0.016s sys 0.065s
I20260807 11:11:17.494568 30937 raft_consensus.cc:504] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:17.459426 30721 consensus_queue.cc:260] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:17.548303 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 72339us
I20260807 11:11:17.550028 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:17.526782 30922 raft_consensus.cc:482] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:17.530920 30940 raft_consensus.cc:2781] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 11:11:17.538908 30171 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 FOLLOWER]: Advancing to term 10
I20260807 11:11:17.556367 25981 leader_election.cc:302] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:17.433961 25921 raft_consensus.cc:2454] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 7.
I20260807 11:11:17.537744 30927 raft_consensus.cc:686] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [term 10 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:11:17.526213 30866 consensus_queue.cc:1035] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.030s
I20260807 11:11:17.568051 25981 leader_election.cc:302] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:17.574013 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:17.574839 25981 leader_election.cc:302] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:17.576589 25981 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:11:17.574829 26938 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.025s sys 0.046s
W20260807 11:11:17.577833 26938 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.615s user 0.025s sys 0.046s
W20260807 11:11:17.549256 26037 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.662s user 0.017s sys 0.066s
I20260807 11:11:17.584472 25981 leader_election.cc:302] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:17.574947 26985 raft_consensus.cc:2454] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 7.
I20260807 11:11:17.560389 30934 raft_consensus.cc:3037] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:11:17.591639 26938 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:11:17.560391 29966 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:17.528008 30881 consensus_queue.cc:237] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:17.567798 30960 raft_consensus.cc:2781] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260807 11:11:17.574967 30940 raft_consensus.cc:686] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 7 LEADER]: Becoming Leader. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:17.600800 26041 raft_consensus.cc:1261] T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 11:11:17.627231 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 544) took 4441 ms. Trace:
I20260807 11:11:17.573843 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:17.592679 30770 raft_consensus.cc:482] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:17.568943 30922 raft_consensus.cc:504] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:17.627681 26041 rpcz_store.cc:276] 0807 11:11:13.186009 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:17.589468 (+4403459us) service_pool.cc:227] Handling call
0807 11:11:17.600326 (+ 10858us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:17.626847 (+ 26521us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:17.627159 (+ 312us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:17.671622 26976 raft_consensus.cc:2454] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 10.
W20260807 11:11:17.631803 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 47402.9us
I20260807 11:11:17.638557 30770 raft_consensus.cc:504] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:17.599025 30475 leader_election.cc:288] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:17.622910 29966 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:17.646603 30398 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:17.646605 30556 raft_consensus.cc:2781] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 11:11:17.596031 30955 raft_consensus.cc:2726] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 11:11:17.625813 30960 raft_consensus.cc:482] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:17.662612 30951 raft_consensus.cc:2726] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 11:11:17.668129 25926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1856) took 3258 ms. Trace:
I20260807 11:11:17.663230 30957 raft_consensus.cc:2726] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 11:11:17.665915 30927 consensus_queue.cc:237] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:17.625851 30923 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:11:17.685949 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 55765.4us
I20260807 11:11:17.704028 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 571) took 1116 ms. Trace:
W20260807 11:11:17.712922 30942 log.cc:924] Time spent T f75da3017fd744719647332808daba49 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.122s user 0.000s sys 0.001s
I20260807 11:11:17.707540 30398 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:17.731127 30957 consensus_queue.cc:1046] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 11:11:17.732213 30398 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:17.715226 30902 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:17.716612 30960 raft_consensus.cc:3037] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 10 FOLLOWER]: Advancing to term 11
I20260807 11:11:17.717226 25926 rpcz_store.cc:276] 0807 11:11:14.409988 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:14.410120 (+ 132us) service_pool.cc:227] Handling call
0807 11:11:14.410372 (+ 252us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:15.193237 (+782865us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:17.639506 (+2446269us) spinlock_profiling.cc:243] Waited 747 ms on lock 0x7b5400407254. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:17.668060 (+ 28554us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1718468864}
I20260807 11:11:17.742218 25926 raft_consensus.cc:1261] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260807 11:11:17.723012 25801 raft_consensus.cc:1226] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 5. Current term is 6. Ops: []
I20260807 11:11:17.722779 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1994) took 1782 ms. Trace:
I20260807 11:11:17.724350 26041 rpcz_store.cc:276] 0807 11:11:16.587590 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:17.675265 (+1087675us) service_pool.cc:227] Handling call
0807 11:11:17.703470 (+ 28205us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:17.703685 (+ 215us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:17.703710 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:17.703757 (+ 47us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:17.703786 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:17.703844 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:17.703966 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:17.732134 30957 consensus_queue.cc:1046] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 11:11:17.709108 30556 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:17.738548 29692 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:17.755474 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:17.758124 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:17.759533 25801 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 3.
W20260807 11:11:17.765004 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 52922.4us
I20260807 11:11:17.722370 30475 raft_consensus.cc:2726] T f31f65eddcdf48f6a64e0a1ddc0c6695 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:17.680827 30946 raft_consensus.cc:504] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:17.753024 30556 raft_consensus.cc:3037] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Advancing to term 9
I20260807 11:11:17.782774 30937 leader_election.cc:288] T 674c5d3caeaf4ccabb23397e655b0e0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:17.762418 30951 raft_consensus.cc:3032] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:11:17.806962 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:17.808257 26041 raft_consensus.cc:2454] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:11:17.736642 30708 raft_consensus.cc:504] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:17.809854 25800 raft_consensus.cc:1226] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:17.753036 30398 consensus_queue.cc:1035] T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.045s
I20260807 11:11:17.753535 30666 consensus_queue.cc:1035] T 3c69ebea3aeb47d48ab8e083fdf71f7c P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20260807 11:11:17.811038 26938 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:17.811072 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:17.811913 25981 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:17.812510 26035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1999) took 1550 ms. Trace:
I20260807 11:11:17.805178 30951 raft_consensus.cc:729] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:11:17.816048 26035 rpcz_store.cc:276] 0807 11:11:16.262169 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:17.467111 (+1204942us) service_pool.cc:227] Handling call
0807 11:11:17.475477 (+ 8366us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:17.475743 (+ 266us) raft_consensus.cc:1520] Early marking committed up to index 5
0807 11:11:17.497803 (+ 22060us) log.cc:842] Serialized 14 byte log entry
0807 11:11:17.812093 (+314290us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:17.812158 (+ 65us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:11:17.812220 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:17.812306 (+ 86us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:17.812447 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":34876,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":55151,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":83,"thread_start_us":313849,"threads_started":2,"wal-append.queue_time_us":287123}
I20260807 11:11:17.821715 25981 leader_election.cc:302] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:17.821630 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:17.733560 30923 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:17.822583 26041 raft_consensus.cc:3037] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Advancing to term 9
I20260807 11:11:17.811009 30475 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:17.784863 29692 consensus_queue.cc:1046] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20260807 11:11:17.832495 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:17.833474 30977 raft_consensus.cc:3032] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 11:11:17.833940 30977 raft_consensus.cc:729] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:11:17.834812 30977 consensus_queue.cc:260] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:17.836146 30977 raft_consensus.cc:3037] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:17.819444 30390 raft_consensus.cc:2781] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 11:11:17.819399 30937 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:17.749351 30940 consensus_queue.cc:237] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:17.813373 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:17.860203 25797 raft_consensus.cc:2454] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 7.
I20260807 11:11:17.866118 26938 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:17.894762 25981 leader_election.cc:302] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:17.894860 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:17.896715 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:17.750172 30902 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:17.905074 30985 raft_consensus.cc:2781] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:17.905705 30985 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:17.906421 30985 raft_consensus.cc:3037] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:17.906965 30957 raft_consensus.cc:2781] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 11:11:17.907821 30957 raft_consensus.cc:686] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [term 7 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:11:17.909176 30957 consensus_queue.cc:237] T 955db4a12dc04bfe8124047fc46823d3 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:17.831583 30927 raft_consensus.cc:2726] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:17.906543 30959 raft_consensus.cc:2726] T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260807 11:11:17.921085 30954 raft_consensus.cc:2726] T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:17.922042 30982 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:17.922705 30982 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:17.927248 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab2949574e4c6badfcf68e51b9c072" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:17.933477 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:17.853456 30390 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:17.906301 30840 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:17.938525 25801 raft_consensus.cc:1226] T d8012276091e4b63b062cb7152208a16 P 606d49e053a346e486f48cb31832b815 [term 5 LEADER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 4. Current term is 5. Ops: []
I20260807 11:11:17.902137 30922 leader_election.cc:288] T cccdf448d32448deae255de6bf7b4125 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:17.859120 30475 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:17.899201 30937 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:17.933650 30951 consensus_queue.cc:260] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:17.938158 30390 raft_consensus.cc:3037] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:17.965902 25981 leader_election.cc:302] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:18.154887 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:11:18.483917 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 518099us
W20260807 11:11:18.464838 26041 consensus_meta.cc:287] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.634s user 0.007s sys 0.009s
I20260807 11:11:18.490139 26041 raft_consensus.cc:2454] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 9.
I20260807 11:11:18.164414 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 401) took 2197 ms. Trace:
I20260807 11:11:18.500164 25932 rpcz_store.cc:276] 0807 11:11:15.966593 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:15.966775 (+ 182us) service_pool.cc:227] Handling call
0807 11:11:15.967167 (+ 392us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:16.963284 (+996117us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:18.164125 (+1200841us) spinlock_profiling.cc:243] Waited 975 ms on lock 0x7b54000361d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:18.164347 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2243197184}
W20260807 11:11:18.163792 25808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1618) took 1721 ms (client timeout 2186 ms). Trace:
I20260807 11:11:18.501168 26938 leader_election.cc:302] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:18.217662 25799 raft_consensus.cc:2454] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 8.
I20260807 11:11:18.502952 30415 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:11:18.503773 30776 consensus_meta.cc:287] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.994s user 0.005s sys 0.008s
W20260807 11:11:18.425704 30988 log.cc:924] Time spent T a09a74fdae5d44c8a9b375e1cfdea1e9 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.104s user 0.001s sys 0.000s
I20260807 11:11:18.504235 30776 raft_consensus.cc:504] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:18.071838 30973 log.cc:924] Time spent T a09a74fdae5d44c8a9b375e1cfdea1e9 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.253s user 0.000s sys 0.000s
I20260807 11:11:18.068976 30922 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:18.503419 30415 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:18.506502 26035 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 4.
I20260807 11:11:18.506345 30922 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:18.507382 30776 leader_election.cc:288] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:18.508177 30776 raft_consensus.cc:2726] T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 11:11:18.505475 30937 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:11:18.074857 30956 log.cc:924] Time spent T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.340s user 0.000s sys 0.001s
W20260807 11:11:18.077945 30990 log.cc:924] Time spent T f40bb008c149418989fdcd7c1e67d554 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.152s user 0.000s sys 0.001s
I20260807 11:11:18.537213 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59312 (request call id 1905) took 3278 ms. Trace:
I20260807 11:11:18.537642 26037 rpcz_store.cc:276] 0807 11:11:15.259289 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:16.786585 (+1527296us) service_pool.cc:227] Handling call
0807 11:11:16.794897 (+ 8312us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:16.814182 (+ 19285us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:16.814231 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:16.837018 (+ 22787us) log.cc:842] Serialized 26 byte log entry
0807 11:11:17.596192 (+759174us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:17.596252 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:17.596350 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:18.536937 (+940587us) raft_consensus.cc:1671] finished
0807 11:11:18.536966 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:18.537134 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":298931,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":9344,"thread_start_us":695733,"threads_started":1,"wal-append.queue_time_us":803134}
I20260807 11:11:18.069525 30556 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:11:18.501731 25808 rpcz_store.cc:269] 0807 11:11:16.442684 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:16.442834 (+ 150us) service_pool.cc:227] Handling call
0807 11:11:18.163540 (+1720706us) spinlock_profiling.cc:243] Waited 325 ms on lock 0x7b540014f5d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:18.163716 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":748200704}
W20260807 11:11:18.226670 30966 log.cc:924] Time spent T 437f2acbd46c43b2834bf00f08d22f65 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 0.478s user 0.001s sys 0.001s
W20260807 11:11:18.073827 30974 log.cc:924] Time spent T 01cf60138c9d41bbb8d914b0a2736151 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.225s user 0.000s sys 0.001s
I20260807 11:11:18.458529 30923 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:18.199595 30987 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:17.948808 30840 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:18.161696 30718 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:18.075876 30982 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:18.502157 29966 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:18.588132 26985 raft_consensus.cc:1226] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 8. Current term is 9. Ops: []
I20260807 11:11:18.540467 30475 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:18.595500 30987 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:18.602072 30840 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:18.593645 30718 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:18.605408 30718 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:11:18.606968 26036 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59324 (request call id 2018) took 1274 ms (client timeout 1684 ms). Trace:
W20260807 11:11:18.607393 26036 rpcz_store.cc:269] 0807 11:11:17.332706 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:17.355057 (+ 22351us) service_pool.cc:227] Handling call
0807 11:11:18.606776 (+1251719us) spinlock_profiling.cc:243] Waited 27.7 ms on lock 0x7b5400229f54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:18.606904 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":63811968}
I20260807 11:11:18.195142 30981 consensus_queue.cc:1046] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.793s
I20260807 11:11:18.622027 30987 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:11:18.627038 30906 log.cc:924] Time spent T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 1.565s user 0.004s sys 0.008s
I20260807 11:11:18.628501 25798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 457) took 3381 ms. Trace:
I20260807 11:11:18.628906 25798 rpcz_store.cc:276] 0807 11:11:15.246694 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:15.246908 (+ 214us) service_pool.cc:227] Handling call
0807 11:11:15.296429 (+ 49521us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:16.158617 (+862188us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:16.158655 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:16.159205 (+ 550us) log.cc:842] Serialized 26 byte log entry
0807 11:11:16.397006 (+237801us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:11:16.397259 (+ 253us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:16.397437 (+ 178us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:18.626265 (+2228828us) spinlock_profiling.cc:243] Waited 9.11 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:11:18.628156 (+ 1891us) raft_consensus.cc:1671] finished
0807 11:11:18.628177 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:18.628438 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":155022,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"spinlock_wait_cycles":20953472,"thread_start_us":1098523,"threads_started":2,"wal-append.queue_time_us":902500}
W20260807 11:11:18.631075 30960 consensus_meta.cc:287] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.868s user 0.010s sys 0.001s
I20260807 11:11:18.631609 30960 raft_consensus.cc:504] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:18.632690 30922 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:18.595619 30927 raft_consensus.cc:482] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
W20260807 11:11:18.633723 30934 consensus_meta.cc:287] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 1.041s user 0.008s sys 0.006s
I20260807 11:11:18.634148 30934 raft_consensus.cc:504] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:18.633935 30927 raft_consensus.cc:504] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:18.636243 30934 leader_election.cc:288] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 4 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:18.641734 26041 raft_consensus.cc:1226] T a805d50b18af4d5eb99f7c96e790e1dd P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 3. Current term is 4. Ops: []
W20260807 11:11:18.642565 30977 consensus_meta.cc:287] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.806s user 0.015s sys 0.000s
W20260807 11:11:18.611711 30795 consensus_meta.cc:287] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 1.135s user 0.016s sys 0.000s
I20260807 11:11:18.643154 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 2004) took 1773 ms. Trace:
I20260807 11:11:18.643522 30795 raft_consensus.cc:2726] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465) with higher term. Message: Invalid argument: T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7.
I20260807 11:11:18.643657 26041 rpcz_store.cc:276] 0807 11:11:16.870144 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:18.492147 (+1622003us) service_pool.cc:227] Handling call
0807 11:11:18.492554 (+ 407us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:18.642239 (+149685us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:18.642892 (+ 653us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b54002de954. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:18.643085 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":342492416}
I20260807 11:11:18.645078 26986 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 527) took 1336 ms. Trace:
I20260807 11:11:18.645530 26986 rpcz_store.cc:276] 0807 11:11:17.308434 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:17.308584 (+ 150us) service_pool.cc:227] Handling call
0807 11:11:17.323337 (+ 14753us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:17.329805 (+ 6468us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:17.329848 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:17.330397 (+ 549us) log.cc:842] Serialized 26 byte log entry
0807 11:11:17.599668 (+269271us) raft_consensus.cc:1631] Marking committed up to 8
0807 11:11:17.599741 (+ 73us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:17.599844 (+ 103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:18.644827 (+1044983us) raft_consensus.cc:1671] finished
0807 11:11:18.644856 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:18.645009 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5974,"raft.queue_time_us":27358,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":16203,"thread_start_us":210258,"threads_started":1,"wal-append.queue_time_us":339548}
I20260807 11:11:18.647616 30794 leader_election.cc:288] T 98fe28118d3d4770aa8de6f8d38ca61f P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:18.654552 30556 leader_election.cc:288] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 9 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:11:18.656483 25926 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.912s user 0.011s sys 0.044s
W20260807 11:11:18.573190 30171 consensus_meta.cc:287] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.992s user 0.013s sys 0.001s
I20260807 11:11:18.542268 30849 raft_consensus.cc:2781] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 FOLLOWER]: Leader election won for term 9
I20260807 11:11:18.663053 30849 raft_consensus.cc:686] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
W20260807 11:11:18.662257 25926 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.918s user 0.012s sys 0.044s
I20260807 11:11:18.550693 30951 raft_consensus.cc:3037] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:11:18.677662 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:18.665298 30981 raft_consensus.cc:482] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:18.690191 26041 raft_consensus.cc:1226] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 11 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 10. Current term is 11. Ops: []
I20260807 11:11:18.690986 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 2011) took 1486 ms. Trace:
I20260807 11:11:18.691383 26041 rpcz_store.cc:276] 0807 11:11:17.205023 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:18.679082 (+1474059us) service_pool.cc:227] Handling call
0807 11:11:18.689958 (+ 10876us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:18.690688 (+ 730us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:18.690922 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:11:18.687644 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 194562us
W20260807 11:11:18.686522 30881 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.761s user 0.019s sys 0.047s
I20260807 11:11:18.701505 26041 raft_consensus.cc:1226] T 2ff0e05310b145358043bba0a9a8ba11 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:18.704305 26986 raft_consensus.cc:1261] T f9465d2caa3e43c8891f52350177df8d P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260807 11:11:18.687821 30927 leader_election.cc:288] T 01cf60138c9d41bbb8d914b0a2736151 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:18.708079 30547 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:18.708204 30960 leader_election.cc:288] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 11 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:18.680552 30849 consensus_queue.cc:237] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:18.709204 30770 leader_election.cc:288] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:18.708750 30547 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:18.694365 30981 raft_consensus.cc:504] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:18.711777 30547 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:18.697345 30864 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:18.714253 30946 leader_election.cc:288] T 6eed5249afac4e16a4b17938ad6e1353 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:18.713724 30864 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:11:18.717270 30878 log.cc:924] Time spent T 0fc5b1dfff1a4556abdd562afbebd768 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 2.291s user 0.012s sys 0.004s
I20260807 11:11:18.719262 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 562) took 3902 ms. Trace:
I20260807 11:11:18.719607 26046 rpcz_store.cc:276] 0807 11:11:14.816501 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:15.746234 (+929733us) service_pool.cc:227] Handling call
0807 11:11:15.751413 (+ 5179us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:15.751904 (+ 491us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:15.751945 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:15.754503 (+ 2558us) log.cc:842] Serialized 26 byte log entry
0807 11:11:16.119178 (+364675us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:16.119242 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:16.119342 (+ 100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:18.716524 (+2597182us) spinlock_profiling.cc:243] Waited 65.6 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:11:18.718797 (+ 2273us) raft_consensus.cc:1671] finished
0807 11:11:18.718819 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:18.719197 (+ 378us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2165,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":9335,"spinlock_wait_cycles":150814464,"thread_start_us":305615,"threads_started":1,"wal-append.queue_time_us":562074}
I20260807 11:11:18.673570 30171 raft_consensus.cc:504] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:11:18.725759 26980 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50960 (request call id 391) took 1887 ms (client timeout 1937 ms). Trace:
W20260807 11:11:18.726307 26980 rpcz_store.cc:269] 0807 11:11:16.847779 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:16.855985 (+ 8206us) service_pool.cc:227] Handling call
0807 11:11:18.725592 (+1869607us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b54006b1554. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:18.725704 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":587987200}
I20260807 11:11:18.728096 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 2013) took 1521 ms. Trace:
W20260807 11:11:18.698729 30881 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.773s user 0.019s sys 0.048s
I20260807 11:11:18.712592 30981 leader_election.cc:288] T dacbdb7bffca4346a1e00e7b1e90fdea P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:11:18.742578 30985 consensus_meta.cc:287] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.835s user 0.013s sys 0.003s
I20260807 11:11:18.742698 30918 raft_consensus.cc:3032] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 11:11:18.695633 30390 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:18.740748 30864 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:18.743136 30985 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:18.744361 30918 raft_consensus.cc:729] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
W20260807 11:11:18.751248 30936 log.cc:924] Time spent T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 1.688s user 0.007s sys 0.007s
W20260807 11:11:18.755177 30909 log.cc:924] Time spent T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 1.814s user 0.001s sys 0.027s
I20260807 11:11:18.755594 30985 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:11:18.757041 30839 log.cc:924] Time spent T 54a291aa09434b06adbb86586dacec0e P eb1fca62dcb34ed9b32188ccb19f0543: Append to log took a long time: real 2.970s user 0.007s sys 0.014s
I20260807 11:11:18.753247 30918 consensus_queue.cc:260] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:11:18.803290 25750 leader_election.cc:334] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Timed out: RequestConsensusVote RPC to 127.23.120.130:36465 timed out after 1.746s (SENT)
I20260807 11:11:18.827085 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:18.831689 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:18.833932 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:18.835207 25798 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 3.
I20260807 11:11:18.837471 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:11:18.840531 25750 leader_election.cc:334] T 740366b034d646e09bc075abffba6286 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Timed out: RequestConsensusVote RPC to 127.23.120.130:36465 timed out after 1.807s (SENT)
W20260807 11:11:18.842928 25750 leader_election.cc:334] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Timed out: RequestConsensusVote RPC to 127.23.120.130:36465 timed out after 1.923s (SENT)
I20260807 11:11:18.844811 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:18.855568 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:18.856412 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:18.857002 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:18.858186 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:18.858727 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:18.860010 25797 raft_consensus.cc:2379] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 5: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:11:18.864986 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1985) took 3611 ms. Trace:
I20260807 11:11:18.728925 26041 rpcz_store.cc:276] 0807 11:11:17.206695 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:18.692775 (+1486080us) service_pool.cc:227] Handling call
0807 11:11:18.701280 (+ 8505us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:18.722543 (+ 21263us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:18.727951 (+ 5408us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:18.865504 26033 rpcz_store.cc:276] 0807 11:11:15.253753 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:15.837112 (+583359us) service_pool.cc:227] Handling call
0807 11:11:15.837540 (+ 428us) raft_consensus.cc:1457] Updating replica for 3 ops
0807 11:11:15.838595 (+ 1055us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:15.838638 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0807 11:11:15.839539 (+ 901us) log.cc:842] Serialized 78 byte log entry
0807 11:11:16.227453 (+387914us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:18.664532 (+2437079us) spinlock_profiling.cc:243] Waited 15.6 ms on lock 0x7b1800072760. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbf35382c 00007f3fbc77e9dc 00007f3fc55b5a39 00007f3fbc79b8bb 00007f3fbc799cbf 00007f3fbc799155 00007f3fc3099167 00007f3fbf6ce807 00007f3fbf6ddf0f
0807 11:11:18.664720 (+ 188us) log.cc:842] Serialized 14 byte log entry
0807 11:11:18.863941 (+199221us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:18.864124 (+ 183us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:18.864205 (+ 81us) raft_consensus.cc:1671] finished
0807 11:11:18.864341 (+ 136us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:18.864692 (+ 351us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":414020,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":153,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":122,"spinlock_wait_cycles":35916672,"thread_start_us":681194,"threads_started":3,"wal-append.queue_time_us":650854,"wal-append.run_cpu_time_us":7541,"wal-append.run_wall_time_us":1591785}
I20260807 11:11:18.868458 26041 raft_consensus.cc:1226] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7. Ops: []
W20260807 11:11:18.871809 26936 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.751s user 0.012s sys 0.036s
I20260807 11:11:18.875844 28964 raft_consensus.cc:482] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:18.876210 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 2020) took 1439 ms. Trace:
I20260807 11:11:18.876945 26041 rpcz_store.cc:276] 0807 11:11:17.437001 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:18.867191 (+1430190us) service_pool.cc:227] Handling call
0807 11:11:18.867861 (+ 670us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:18.875790 (+ 7929us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:18.876084 (+ 294us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:18.876909 28964 raft_consensus.cc:504] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:18.879053 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:18.881033 26046 raft_consensus.cc:1261] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 11:11:18.882095 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 2019) took 1467 ms. Trace:
W20260807 11:11:18.882114 26936 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.761s user 0.013s sys 0.037s
I20260807 11:11:18.882570 26046 rpcz_store.cc:276] 0807 11:11:17.414728 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:18.721222 (+1306494us) service_pool.cc:227] Handling call
0807 11:11:18.721548 (+ 326us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:18.881623 (+160075us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:18.882020 (+ 397us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:18.887784 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
W20260807 11:11:18.874627 26034 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.190s user 0.015s sys 0.043s
W20260807 11:11:18.889118 26034 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.204s user 0.015s sys 0.043s
I20260807 11:11:18.893520 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:18.902372 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 2021) took 1360 ms. Trace:
I20260807 11:11:18.902268 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986fe968b7be4eab9c480ec85c0384b0" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:18.902896 26033 rpcz_store.cc:276] 0807 11:11:17.541758 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:18.868435 (+1326677us) service_pool.cc:227] Handling call
0807 11:11:18.901297 (+ 32862us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:18.901987 (+ 690us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:18.902011 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:18.902062 (+ 51us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:18.902091 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:18.902163 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:18.902300 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:18.904150 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 574) took 1161 ms. Trace:
I20260807 11:11:18.904415 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:18.911908 29991 raft_consensus.cc:482] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:18.912612 25798 raft_consensus.cc:1226] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 5. Current term is 6. Ops: []
I20260807 11:11:18.913094 29991 raft_consensus.cc:504] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:18.915091 30390 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 8 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:18.912133 29851 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:18.916729 30171 leader_election.cc:288] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 10 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:18.920641 28964 leader_election.cc:288] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:18.917354 30718 consensus_queue.cc:1035] T f9465d2caa3e43c8891f52350177df8d P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.005s
I20260807 11:11:18.930070 25796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53312 (request call id 1937) took 2789 ms. Trace:
I20260807 11:11:18.930312 26033 raft_consensus.cc:1226] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 4. Current term is 5. Ops: []
I20260807 11:11:18.930972 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 576) took 1177 ms. Trace:
I20260807 11:11:18.931375 26033 rpcz_store.cc:276] 0807 11:11:17.753346 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:18.929343 (+1175997us) service_pool.cc:227] Handling call
0807 11:11:18.929978 (+ 635us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:18.930729 (+ 751us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:18.930902 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 11:11:18.932017 25799 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1621) took 2474 ms (client timeout 1574 ms). Trace:
W20260807 11:11:18.932464 25799 rpcz_store.cc:269] 0807 11:11:16.457505 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:16.457691 (+ 186us) service_pool.cc:227] Handling call
0807 11:11:18.931647 (+2473956us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54000474d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:18.931834 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2770004352}
I20260807 11:11:18.934631 25800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 466) took 2125 ms. Trace:
I20260807 11:11:18.948272 25800 rpcz_store.cc:276] 0807 11:11:16.809498 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:17.217966 (+408468us) service_pool.cc:227] Handling call
0807 11:11:17.230141 (+ 12175us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:17.812381 (+582240us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:18.934110 (+1121729us) spinlock_profiling.cc:243] Waited 579 ms on lock 0x7b54003ad4d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:18.934433 (+ 323us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1332596992}
I20260807 11:11:18.920203 29851 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:18.953960 30475 raft_consensus.cc:3032] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 11:11:18.954646 30475 raft_consensus.cc:729] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:11:18.904920 26041 rpcz_store.cc:276] 0807 11:11:17.742720 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:18.902516 (+1159796us) service_pool.cc:227] Handling call
0807 11:11:18.903130 (+ 614us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:18.903653 (+ 523us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:18.903689 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:18.903772 (+ 83us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:18.903848 (+ 76us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:18.903915 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:18.904049 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:18.936092 25801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 476) took 1209 ms. Trace:
W20260807 11:11:18.937181 25806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53324 (request call id 1625) took 2427 ms (client timeout 2247 ms). Trace:
W20260807 11:11:18.945552 26976 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50942 (request call id 513) took 5051 ms (client timeout 2007 ms). Trace:
I20260807 11:11:18.956486 29851 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:18.943673 26035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43246 (request call id 579) took 1168 ms. Trace:
I20260807 11:11:18.930570 25796 rpcz_store.cc:276] 0807 11:11:16.141081 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:16.141243 (+ 162us) service_pool.cc:227] Handling call
0807 11:11:16.141581 (+ 338us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:17.326407 (+1184826us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:18.929467 (+1603060us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b5400137254. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:18.929853 (+ 386us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2722692992}
I20260807 11:11:18.962006 25801 rpcz_store.cc:276] 0807 11:11:17.726200 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:17.760390 (+ 34190us) service_pool.cc:227] Handling call
0807 11:11:17.760745 (+ 355us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:17.945962 (+185217us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:18.935530 (+989568us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b54003acfd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:18.935887 (+ 357us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":408178560}
W20260807 11:11:18.963456 25806 rpcz_store.cc:269] 0807 11:11:16.509287 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:16.744098 (+234811us) service_pool.cc:227] Handling call
0807 11:11:18.936685 (+2192587us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b54000e51d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:18.936933 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":464654720}
I20260807 11:11:18.918361 30795 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:18.976872 30795 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:18.978682 25797 raft_consensus.cc:2379] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 7: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:11:18.980839 30922 raft_consensus.cc:482] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:18.981962 30721 raft_consensus.cc:3037] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:18.981803 30922 raft_consensus.cc:504] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:18.918004 30794 consensus_queue.cc:1046] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.066s
I20260807 11:11:18.984633 30794 raft_consensus.cc:3032] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 8 LEADER]: Stepping down as leader of term 8
I20260807 11:11:18.985069 30794 raft_consensus.cc:729] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:18.986426 30780 raft_consensus.cc:3032] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 11:11:18.986943 30780 raft_consensus.cc:729] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:18.967305 26035 rpcz_store.cc:276] 0807 11:11:17.775422 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:17.866014 (+ 90592us) service_pool.cc:227] Handling call
0807 11:11:18.943142 (+1077128us) spinlock_profiling.cc:243] Waited 572 ms on lock 0x7b5400373cd4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:18.943399 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1316031872}
I20260807 11:11:18.988516 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:18.989526 26035 raft_consensus.cc:3037] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:18.994451 30927 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:18.995533 30927 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:18.996637 26938 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:18.986469 30794 consensus_queue.cc:260] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:19.000116 30794 raft_consensus.cc:3037] T 986fe968b7be4eab9c480ec85c0384b0 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Advancing to term 9
I20260807 11:11:19.002249 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:11:18.965109 26976 rpcz_store.cc:269] 0807 11:11:13.893468 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:13.893619 (+ 151us) service_pool.cc:227] Handling call
0807 11:11:15.308986 (+1415367us) spinlock_profiling.cc:243] Waited 4.78 ms on lock 0x7b5800000f98. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc764851 00007f3fbc764702 00007f3fc5654b09 00007f3fc4590730 00007f3fc465679b 00007f3fc4656821 00007f3fc46061b0 00007f3fc4605a94 00007f3fbdcedc04 00007f3fbdcedb92
0807 11:11:18.945032 (+3636046us) spinlock_profiling.cc:243] Waited 2.3 s on lock 0x7b54006b26d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:18.945299 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5312057472}
I20260807 11:11:18.988457 30780 consensus_queue.cc:260] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:19.010861 30780 raft_consensus.cc:3037] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:19.014163 25786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59118 (request call id 464) took 2205 ms. Trace:
I20260807 11:11:19.014580 30795 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:11:19.015298 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 47594.9us
I20260807 11:11:17.744796 26038 rpcz_store.cc:276] 0807 11:11:15.940573 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:17.378826 (+1438253us) service_pool.cc:227] Handling call
0807 11:11:17.379116 (+ 290us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:17.379327 (+ 211us) raft_consensus.cc:1520] Early marking committed up to index 5
0807 11:11:17.396484 (+ 17157us) log.cc:842] Serialized 14 byte log entry
0807 11:11:17.722463 (+325979us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:17.722513 (+ 50us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:11:17.722563 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:17.722622 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:17.722728 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":16980,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":3290,"raft.queue_time_us":123979,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":3115,"thread_start_us":325484,"threads_started":2,"wal-append.queue_time_us":317502}
I20260807 11:11:18.956290 30475 consensus_queue.cc:260] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:19.029177 30972 log.cc:924] Time spent T cb81660c5a8b4365a928d2ef1e4d0be5 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 1.267s user 0.014s sys 0.010s
I20260807 11:11:19.030212 30475 raft_consensus.cc:3037] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:19.030844 30421 raft_consensus.cc:2726] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:19.024792 30952 raft_consensus.cc:3032] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 11:11:19.031898 30952 raft_consensus.cc:729] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:19.026003 30415 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:19.028116 30922 leader_election.cc:288] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:19.034907 30421 raft_consensus.cc:482] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:19.035748 30421 raft_consensus.cc:504] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:19.041252 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 2.56168e+06us
I20260807 11:11:19.042891 30918 raft_consensus.cc:3037] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:19.041918 25655 catalog_manager.cc:5495] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc reported cstate change: term changed from 2 to 5, leader changed from eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132) to 4fd6381544574ddca559ce63b9243efc (127.23.120.130). New cstate: current_term: 5 leader_uuid: "4fd6381544574ddca559ce63b9243efc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 11:11:19.045073 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:19.046195 26986 raft_consensus.cc:2454] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:11:19.023654 30630 raft_consensus.cc:2781] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 11:11:19.047274 30630 raft_consensus.cc:482] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:19.030295 30927 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:19.030997 25920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1863) took 3989 ms. Trace:
I20260807 11:11:19.019935 30876 consensus_queue.cc:1046] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 11:11:19.007841 26033 raft_consensus.cc:1261] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260807 11:11:19.014634 25786 rpcz_store.cc:276] 0807 11:11:16.808717 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:17.009575 (+200858us) service_pool.cc:227] Handling call
0807 11:11:17.039428 (+ 29853us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:17.134230 (+ 94802us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:17.134269 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:17.134311 (+ 42us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:17.134358 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:19.013608 (+1879250us) spinlock_profiling.cc:243] Waited 94.5 ms on lock 0x7b54003acad4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:19.013677 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:19.013989 (+ 312us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":217458944}
I20260807 11:11:19.021371 30749 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:19.047785 30630 raft_consensus.cc:3037] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:11:19.057839 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 577) took 1290 ms. Trace:
I20260807 11:11:19.062278 26033 rpcz_store.cc:276] 0807 11:11:17.763427 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:18.932219 (+1168792us) service_pool.cc:227] Handling call
0807 11:11:18.932727 (+ 508us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:19.051676 (+118949us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:19.053559 (+ 1883us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:19.068933 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:19.071856 25872 leader_election.cc:302] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:19.074342 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 582) took 1142 ms. Trace:
I20260807 11:11:19.074837 26038 rpcz_store.cc:276] 0807 11:11:17.932284 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:19.064606 (+1132322us) service_pool.cc:227] Handling call
0807 11:11:19.073340 (+ 8734us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:19.073797 (+ 457us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:19.073822 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:19.073862 (+ 40us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:19.073907 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:19.073981 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:19.074108 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:16.034777 24034 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 11:11:19.064198 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:19.086485 30421 leader_election.cc:288] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:19.087425 30415 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:19.087770 30749 raft_consensus.cc:504] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:19.088222 30415 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:19.090179 30749 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:19.090902 30415 leader_election.cc:288] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:19.094224 26038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:59300 (request call id 828) took 7980 ms. Trace:
I20260807 11:11:19.094957 26038 rpcz_store.cc:276] 0807 11:11:11.114110 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:19.085985 (+7971875us) service_pool.cc:227] Handling call
0807 11:11:19.094081 (+ 8096us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 11:11:19.096208 30811 consensus_queue.cc:1035] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:11:19.096721 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:11:19.101514 25654 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:40730 (request call id 235) took 42369 ms (client timeout 15000 ms). Trace:
I20260807 11:11:19.103495 30681 raft_consensus.cc:3037] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Advancing to term 9
I20260807 11:11:19.090283 25981 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
I20260807 11:11:19.107142 30927 raft_consensus.cc:2726] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 11:11:19.090842 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:19.102453 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:19.129029 25750 leader_election.cc:302] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:19.049486 25920 rpcz_store.cc:276] 0807 11:11:15.041603 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:15.070669 (+ 29066us) service_pool.cc:227] Handling call
0807 11:11:15.137800 (+ 67131us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:15.138049 (+ 249us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 11:11:15.312098 (+174049us) log.cc:842] Serialized 14 byte log entry
0807 11:11:15.996362 (+684264us) log.cc:842] Serialized 14 byte log entry
0807 11:11:18.704052 (+2707690us) spinlock_profiling.cc:243] Waited 6.41 ms on lock 0x7b180049e0a0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbc79c638 00007f3fbc79fb11 00007f3fbc79fac9 00007f3fbc79fa59 00007f3fbc79fa21 00007f3fbc79ed1d 00007f3fc5a00204 00007f3fc5a00039 00007f3fbc77edc6
0807 11:11:19.029704 (+325652us) spinlock_profiling.cc:243] Waited 4.9 ms on lock 0x7b180049e0a0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbc7134ca 00007f3fc5df4f99 00007f3fbc5d7990 00007f3fbc799864 00007f3fbc799155 00007f3fbf6bc3d0 00007f3fbf6ce576 00007f3fbf6df0d7 00007f3fbf6df058
0807 11:11:19.030151 (+ 447us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:19.030239 (+ 88us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:19.030390 (+ 151us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:19.030545 (+ 155us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:19.030710 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17329,"prepare.queue_time_us":301614,"prepare.run_cpu_time_us":102,"prepare.run_wall_time_us":105,"raft.queue_time_us":3056944,"raft.run_cpu_time_us":369,"raft.run_wall_time_us":370,"spinlock_wait_cycles":26026112,"thread_start_us":836374,"threads_started":2,"wal-append.queue_time_us":849694,"wal-append.run_cpu_time_us":7318,"wal-append.run_wall_time_us":1727866}
W20260807 11:11:19.101948 25654 rpcz_store.cc:269] 0807 11:10:36.732384 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:10:36.732921 (+ 537us) service_pool.cc:227] Handling call
0807 11:11:19.101206 (+42368285us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 11:10:48.981329 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 11:10:49.268079 (+286750us) write_op.cc:432] Acquiring schema lock in shared mode
0807 11:10:49.272647 (+ 4568us) write_op.cc:435] Acquired schema lock
0807 11:10:49.272685 (+ 38us) tablet.cc:660] Decoding operations
0807 11:10:49.297657 (+ 24972us) write_op.cc:620] Acquiring the partition lock for write op
0807 11:10:49.297768 (+ 111us) write_op.cc:641] Partition lock acquired for write op
0807 11:10:49.297796 (+ 28us) tablet.cc:683] Acquiring locks for 23 operations
0807 11:10:49.299816 (+ 2020us) tablet.cc:699] Row locks acquired
0807 11:10:49.299866 (+ 50us) write_op.cc:260] PREPARE: finished
0807 11:10:49.300071 (+ 205us) write_op.cc:270] Start()
0807 11:10:49.300252 (+ 181us) write_op.cc:276] Timestamp: P: 1786101049299994 usec, L: 0
0807 11:10:49.300297 (+ 45us) op_driver.cc:331] REPLICATION: starting
0807 11:10:49.401239 (+100942us) log.cc:842] Serialized 9186 byte log entry
0807 11:10:53.496238 (+4094999us) spinlock_profiling.cc:243] Waited 5.19 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:10:54.521088 (+1024850us) op_driver.cc:444] REPLICATION: finished
0807 11:10:55.339050 (+817962us) write_op.cc:301] APPLY: starting
0807 11:10:55.364832 (+ 25782us) tablet.cc:1365] starting BulkCheckPresence
0807 11:10:55.551428 (+186596us) tablet.cc:1368] finished BulkCheckPresence
0807 11:10:55.554724 (+ 3296us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 11:10:56.416213 (+861489us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 11:10:56.634264 (+218051us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=23
0807 11:10:56.634315 (+ 51us) write_op.cc:312] APPLY: finished
0807 11:10:56.634990 (+ 675us) log.cc:842] Serialized 158 byte log entry
0807 11:10:58.078697 (+1443707us) write_op.cc:489] Releasing partition, row and schema locks
0807 11:10:58.079229 (+ 532us) write_op.cc:454] Released schema lock
0807 11:10:58.079422 (+ 193us) write_op.cc:341] FINISH: Updating metrics
0807 11:10:58.080101 (+ 679us) write_op.cc:489] Releasing partition, row and schema locks
0807 11:10:58.080161 (+ 60us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":42096,"reported_tablets":39,"child_traces":[["op",{"apply.queue_time_us":641071,"apply.run_cpu_time_us":42126,"apply.run_wall_time_us":2812534,"num_ops":23,"prepare.queue_time_us":288333,"prepare.run_cpu_time_us":57122,"prepare.run_wall_time_us":1763590,"raft.queue_time_us":407863,"raft.run_cpu_time_us":54122,"raft.run_wall_time_us":718171,"replication_time_us":5159568,"spinlock_wait_cycles":11933440,"thread_start_us":3191864,"threads_started":5,"wal-append.queue_time_us":2727209,"wal-append.run_cpu_time_us":71987,"wal-append.run_wall_time_us":5780168}]]}
I20260807 11:11:19.136885 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:19.137856 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437f2acbd46c43b2834bf00f08d22f65" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:19.139343 26979 raft_consensus.cc:1261] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 10 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260807 11:11:19.139716 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:19.140246 30902 leader_election.cc:288] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:19.147312 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:19.150669 30954 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:19.151448 30954 raft_consensus.cc:504] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:19.154544 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:19.155596 25801 raft_consensus.cc:2379] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 8: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:11:19.142520 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:19.157689 26984 raft_consensus.cc:2354] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 7. Current term is 8.
I20260807 11:11:16.157234 26042 rpcz_store.cc:276] 0807 11:11:13.203161 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:14.718243 (+1515082us) service_pool.cc:227] Handling call
0807 11:11:14.730600 (+ 12357us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:14.730844 (+ 244us) raft_consensus.cc:1520] Early marking committed up to index 6
0807 11:11:14.768624 (+ 37780us) log.cc:842] Serialized 14 byte log entry
0807 11:11:15.809894 (+1041270us) spinlock_profiling.cc:243] Waited 3.64 ms on lock 0x7b1800029ce0. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbc77e345 00007f3fc55b5a39 00007f3fbc79b8bb 00007f3fbc799cbf 00007f3fbc79be22 00007f3fbf3455bb 00007f3fbf34b063 00007f3fbf34b57b 00007f3fbf6ce88a
0807 11:11:16.156185 (+346291us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:16.156256 (+ 71us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:16.156337 (+ 81us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:16.156435 (+ 98us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:16.156580 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23748,"prepare.queue_time_us":37572,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":9622,"raft.queue_time_us":5656,"spinlock_wait_cycles":8369920,"thread_start_us":346005,"threads_started":1}
I20260807 11:11:19.135481 30749 raft_consensus.cc:2781] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 11:11:19.159685 30749 raft_consensus.cc:482] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:19.160105 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:19.161195 26936 leader_election.cc:302] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:19.160125 30749 raft_consensus.cc:3037] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:19.172025 25801 raft_consensus.cc:1226] T e0766ccf18b84e41ac68eb6599a834c5 P 606d49e053a346e486f48cb31832b815 [term 7 LEADER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:19.175923 30955 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:19.177971 26979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:19.181280 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:19.183800 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:19.184634 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:19.185642 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:19.186350 26984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fb619c2ada475f817fbcb9739a401a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:19.144200 26978 raft_consensus.cc:2362] T b7c44ad3ac214b2d83c41bc5aa01f349 P eb1fca62dcb34ed9b32188ccb19f0543 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 10. Re-sending same reply.
I20260807 11:11:19.189116 26039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 1997) took 3247 ms. Trace:
I20260807 11:11:19.191372 30708 leader_election.cc:288] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:19.195508 29847 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:19.196174 26036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 2009) took 2053 ms. Trace:
I20260807 11:11:19.201148 30256 raft_consensus.cc:482] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:19.201776 26037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59324 (request call id 2008) took 2059 ms. Trace:
I20260807 11:11:19.202199 30278 raft_consensus.cc:482] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:11:19.206081 25916 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1665) took 2726 ms (client timeout 2144 ms). Trace:
I20260807 11:11:19.278047 26987 raft_consensus.cc:1261] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260807 11:11:19.211050 25639 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 66d10b6b12404f0488b9f83c8080069f with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica b22d69db45194908ad3d345eba9badd9 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:11:19.278766 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:19.215811 30421 raft_consensus.cc:2781] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260807 11:11:19.216269 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 161929us
I20260807 11:11:19.279557 30421 raft_consensus.cc:482] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting leader election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:19.280119 30421 raft_consensus.cc:3037] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Advancing to term 9
I20260807 11:11:19.280304 25872 leader_election.cc:302] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:11:19.216405 26035 raft_consensus.cc:2454] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 8.
I20260807 11:11:19.282725 25796 raft_consensus.cc:2354] T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 7. Current term is 8.
I20260807 11:11:19.271620 25799 raft_consensus.cc:2454] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 6.
W20260807 11:11:19.207345 25921 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39752 (request call id 1659) took 2762 ms (client timeout 2186 ms). Trace:
W20260807 11:11:19.278918 25639 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 68118.1us
I20260807 11:11:19.213445 30852 raft_consensus.cc:686] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [term 7 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:19.324333 25786 raft_consensus.cc:2379] T 99e5feeda1eb4affb2532bda1b0055cd P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 6: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
W20260807 11:11:19.277654 25916 rpcz_store.cc:269] 0807 11:11:16.479850 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:16.777060 (+297210us) service_pool.cc:227] Handling call
0807 11:11:19.205567 (+2428507us) spinlock_profiling.cc:243] Waited 285 ms on lock 0x7b54000532d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:19.205787 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":654510336}
I20260807 11:11:19.282001 25800 raft_consensus.cc:2362] T dee1fb79ff3944c08f589c435f2f09a9 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 7. Re-sending same reply.
W20260807 11:11:19.284627 25921 rpcz_store.cc:269] 0807 11:11:16.444601 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:16.785615 (+341014us) service_pool.cc:227] Handling call
0807 11:11:19.206890 (+2421275us) spinlock_profiling.cc:243] Waited 445 ms on lock 0x7b5400186354. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:19.207117 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1023070976}
I20260807 11:11:19.341830 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:19.340896 26938 leader_election.cc:302] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:19.268349 26037 rpcz_store.cc:276] 0807 11:11:17.142296 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:18.539245 (+1396949us) service_pool.cc:227] Handling call
0807 11:11:18.539547 (+ 302us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:18.865005 (+325458us) raft_consensus.cc:1520] Early marking committed up to index 8
0807 11:11:18.865578 (+ 573us) log.cc:842] Serialized 14 byte log entry
0807 11:11:19.064380 (+198802us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:19.064453 (+ 73us) raft_consensus.cc:1631] Marking committed up to 8
0807 11:11:19.068673 (+ 4220us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:19.200572 (+131899us) spinlock_profiling.cc:243] Waited 8.2 ms on lock 0x7b180024eb00. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fc563b8b1 00007f3fc5648661 00007f3fbc769f96 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9
0807 11:11:19.201336 (+ 764us) spinlock_profiling.cc:243] Waited 325 ms on lock 0x7b540022a454. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:19.201394 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:19.201584 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7863,"prepare.queue_time_us":102098,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":55,"raft.queue_time_us":15073,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":7056,"spinlock_wait_cycles":765667840,"thread_start_us":183076,"threads_started":1,"wal-append.queue_time_us":409940}
I20260807 11:11:19.257933 26039 rpcz_store.cc:276] 0807 11:11:15.941691 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:17.403756 (+1462065us) service_pool.cc:227] Handling call
0807 11:11:17.417573 (+ 13817us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:17.418022 (+ 449us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:17.418060 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:17.438952 (+ 20892us) log.cc:842] Serialized 26 byte log entry
0807 11:11:17.595501 (+156549us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:11:17.595543 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:17.595626 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:19.028135 (+1432509us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:11:19.127689 (+ 99554us) raft_consensus.cc:1671] finished
0807 11:11:19.127709 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:19.188869 (+ 61160us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":110,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":8110,"spinlock_wait_cycles":26480896,"thread_start_us":129140,"threads_started":1,"wal-append.queue_time_us":253528}
I20260807 11:11:19.354813 26986 raft_consensus.cc:2454] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 10.
I20260807 11:11:19.338662 30708 raft_consensus.cc:482] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:19.364871 30650 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:11:19.365839 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 85736.5us
I20260807 11:11:19.266517 26036 rpcz_store.cc:276] 0807 11:11:17.142652 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:18.608373 (+1465721us) service_pool.cc:227] Handling call
0807 11:11:18.608703 (+ 330us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:18.636078 (+ 27375us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 11:11:18.863713 (+227635us) log.cc:842] Serialized 14 byte log entry
0807 11:11:18.978136 (+114423us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:18.978249 (+ 113us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:18.978349 (+ 100us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:19.195586 (+217237us) spinlock_profiling.cc:243] Waited 27.1 ms on lock 0x7b54002df854. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:19.195659 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:19.195874 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6919,"prepare.queue_time_us":111881,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":2923,"raft.queue_time_us":7182,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":245,"spinlock_wait_cycles":62345600,"thread_start_us":331774,"threads_started":2,"wal-append.queue_time_us":274597}
I20260807 11:11:19.365290 30987 raft_consensus.cc:482] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:19.365954 30954 leader_election.cc:288] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:19.379881 30981 raft_consensus.cc:482] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:19.411877 30700 raft_consensus.cc:2726] T 986fe968b7be4eab9c480ec85c0384b0 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260807 11:11:19.412348 25806 raft_consensus.cc:1226] T 61cb005020884d87b17995a28b8effd6 P 606d49e053a346e486f48cb31832b815 [term 7 LEADER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:19.373839 29847 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:19.413152 30630 raft_consensus.cc:504] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:19.415132 29847 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:19.407505 30681 raft_consensus.cc:2781] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 11:11:19.380200 30650 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:19.426237 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:19.428567 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:19.430616 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:11:19.440340 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 56248us
I20260807 11:11:19.417239 26974 raft_consensus.cc:2379] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 5: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:11:19.356877 30955 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:19.447072 26046 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.007s sys 0.009s
I20260807 11:11:19.370864 30278 raft_consensus.cc:504] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
W20260807 11:11:19.451392 26046 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.542s user 0.007s sys 0.009s
I20260807 11:11:19.416656 30630 leader_election.cc:288] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:19.431881 30997 raft_consensus.cc:482] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
W20260807 11:11:19.459918 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 298906us
I20260807 11:11:19.368310 30256 raft_consensus.cc:504] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:19.421623 30171 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:19.442479 30681 raft_consensus.cc:686] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
W20260807 11:11:19.471956 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.132:43279, user_credentials={real_user=slave}} blocked reactor thread for 227646us
I20260807 11:11:19.472287 26036 raft_consensus.cc:3037] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 4 FOLLOWER]: Advancing to term 5
I20260807 11:11:19.474839 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:19.432101 30977 raft_consensus.cc:482] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:19.391065 30708 raft_consensus.cc:504] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:19.433564 30852 consensus_queue.cc:237] T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:19.478749 30955 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:19.473174 25750 leader_election.cc:302] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 606d49e053a346e486f48cb31832b815; no voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:19.479964 30256 leader_election.cc:288] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:19.480729 30171 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:19.467147 30981 raft_consensus.cc:504] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:19.434010 30987 raft_consensus.cc:504] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:19.485917 30681 consensus_queue.cc:237] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:19.489001 30997 raft_consensus.cc:504] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:19.495025 30278 leader_election.cc:288] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:19.453327 30749 raft_consensus.cc:504] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:19.497890 26985 raft_consensus.cc:2454] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:11:19.517196 30977 raft_consensus.cc:504] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:19.519991 30749 leader_election.cc:288] T 998ef34204b64d2481698cc29e055cc2 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:19.522326 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:19.523764 26042 raft_consensus.cc:2396] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260807 11:11:19.504256 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:11:19.504932 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 53006.7us
W20260807 11:11:19.496620 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 116574us
W20260807 11:11:19.509392 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 44357.4us
I20260807 11:11:19.507985 26041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 581) took 1661 ms. Trace:
I20260807 11:11:19.753008 26041 rpcz_store.cc:276] 0807 11:11:17.846490 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:18.968812 (+1122322us) service_pool.cc:227] Handling call
0807 11:11:19.072910 (+104098us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:19.073782 (+ 872us) raft_consensus.cc:1520] Early marking committed up to index 6
0807 11:11:19.073813 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:19.074515 (+ 702us) log.cc:842] Serialized 26 byte log entry
0807 11:11:19.344646 (+270131us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:19.344792 (+ 146us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:19.344942 (+ 150us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:19.433289 (+ 88347us) spinlock_profiling.cc:243] Waited 18.3 ms on lock 0x7b44000232cc. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc6aea95 00007f3fbc6af4b2 00007f3fbf357511 00007f3fbf352e1a 00007f3fbf346bee 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551 00007f3fbf34f509 00007f3fbf34f499
0807 11:11:19.498362 (+ 65073us) raft_consensus.cc:1671] finished
0807 11:11:19.498385 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:19.507921 (+ 9536us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1899,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":43805,"spinlock_wait_cycles":42132224,"thread_start_us":261633,"threads_started":1,"wal-append.queue_time_us":336411}
I20260807 11:11:19.755384 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:19.755515 30949 consensus_queue.cc:1046] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 11:11:19.756215 26041 raft_consensus.cc:3037] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 8
I20260807 11:11:19.526198 26035 raft_consensus.cc:2396] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 8 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 2.
W20260807 11:11:19.529961 25869 connection.cc:232] RPC call timeout handler was delayed by -1.6700580979231745s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15410000000000001s for a response
I20260807 11:11:19.757045 30949 raft_consensus.cc:3032] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 LEADER]: Stepping down as leader of term 6
W20260807 11:11:19.757308 25869 connection.cc:232] RPC call timeout handler was delayed by -1.1417580980341882s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21280000000000002s for a response
I20260807 11:11:19.757619 30949 raft_consensus.cc:729] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:19.758971 30949 consensus_queue.cc:260] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:19.530072 29847 raft_consensus.cc:482] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:19.760396 29847 raft_consensus.cc:504] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:19.762861 29847 leader_election.cc:288] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:19.530484 31034 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:19.763705 31034 raft_consensus.cc:504] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:19.539582 25981 leader_election.cc:302] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:11:19.766909 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 227412us
I20260807 11:11:19.767741 25981 leader_election.cc:302] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:19.540125 25925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1877) took 2688 ms. Trace:
I20260807 11:11:19.542162 30981 leader_election.cc:288] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:19.769536 30981 raft_consensus.cc:2781] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:19.769868 30922 raft_consensus.cc:2781] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 11:11:19.770079 30981 raft_consensus.cc:482] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
W20260807 11:11:19.770392 31039 log.cc:924] Time spent T 3c69ebea3aeb47d48ab8e083fdf71f7c P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.121s user 0.000s sys 0.003s
I20260807 11:11:19.770653 30981 raft_consensus.cc:3037] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:19.766152 31034 leader_election.cc:288] T 0e15f851c3b34d9893d8a5c89e0f98ff P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:19.531405 30256 raft_consensus.cc:482] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:19.539179 30421 raft_consensus.cc:504] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:19.773123 25981 leader_election.cc:398] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195) with higher term. Message: Invalid argument: T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 7. Current term is 8.
I20260807 11:11:19.773558 25981 leader_election.cc:401] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260807 11:11:19.773417 26041 raft_consensus.cc:2454] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 8.
I20260807 11:11:19.545961 26037 raft_consensus.cc:2454] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 5.
I20260807 11:11:19.546494 30987 leader_election.cc:288] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:19.811080 30987 raft_consensus.cc:482] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:19.546772 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:19.547008 25919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 395) took 4287 ms. Trace:
I20260807 11:11:19.812021 30987 raft_consensus.cc:504] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:19.547606 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:19.813405 25796 raft_consensus.cc:2396] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 11:11:19.548358 31023 raft_consensus.cc:482] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:19.549479 26034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 570) took 2996 ms. Trace:
I20260807 11:11:19.813020 25919 rpcz_store.cc:276] 0807 11:11:15.259170 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:15.803113 (+543943us) service_pool.cc:227] Handling call
0807 11:11:15.803467 (+ 354us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 11:11:15.942634 (+139167us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:15.942679 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 11:11:15.943379 (+ 700us) log.cc:842] Serialized 52 byte log entry
0807 11:11:16.154264 (+210885us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:16.240807 (+ 86543us) log.cc:842] Serialized 14 byte log entry
0807 11:11:19.534478 (+3293671us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:19.546425 (+ 11947us) spinlock_profiling.cc:243] Waited 56.3 ms on lock 0x7b5400052b54. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:19.546468 (+ 43us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:19.546554 (+ 86us) raft_consensus.cc:1671] finished
0807 11:11:19.546660 (+ 106us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:19.546821 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2937593,"prepare.queue_time_us":33856,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":68,"raft.queue_time_us":3046772,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":173,"spinlock_wait_cycles":129516416,"thread_start_us":558017,"threads_started":2,"wal-append.queue_time_us":353686,"wal-append.run_cpu_time_us":2290,"wal-append.run_wall_time_us":1016394}
I20260807 11:11:19.815104 30987 leader_election.cc:288] T 740366b034d646e09bc075abffba6286 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:19.549810 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eed5249afac4e16a4b17938ad6e1353" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:19.816206 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:19.814774 26034 rpcz_store.cc:276] 0807 11:11:16.552845 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:17.513169 (+960324us) service_pool.cc:227] Handling call
0807 11:11:17.515098 (+ 1929us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:17.575815 (+ 60717us) raft_consensus.cc:1520] Early marking committed up to index 8
0807 11:11:17.593176 (+ 17361us) log.cc:842] Serialized 14 byte log entry
0807 11:11:18.889568 (+1296392us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:18.889689 (+ 121us) raft_consensus.cc:1631] Marking committed up to 8
0807 11:11:18.889794 (+ 105us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:19.549040 (+659246us) spinlock_profiling.cc:243] Waited 60.5 ms on lock 0x7b54000bb154. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:19.549086 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:19.549421 (+ 335us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":17664,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":2529,"raft.queue_time_us":47940,"raft.run_cpu_time_us":67,"raft.run_wall_time_us":67,"spinlock_wait_cycles":139082624,"thread_start_us":1293261,"threads_started":2,"wal-append.queue_time_us":232663,"wal-append.run_cpu_time_us":1430,"wal-append.run_wall_time_us":1134878}
I20260807 11:11:19.817152 26038 raft_consensus.cc:1261] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260807 11:11:19.549633 30997 leader_election.cc:288] T 76dbf7480b434c0ba6a902c817cf5455 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:19.551100 30927 raft_consensus.cc:482] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:19.555703 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:19.818900 26038 raft_consensus.cc:1261] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 11:11:19.550441 29991 leader_election.cc:288] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:19.532061 30977 leader_election.cc:288] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:19.819319 30927 raft_consensus.cc:504] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:19.560209 30650 leader_election.cc:288] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:19.563593 30794 raft_consensus.cc:2726] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 11:11:19.820113 25872 leader_election.cc:302] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9
I20260807 11:11:19.820981 25786 raft_consensus.cc:2454] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 5.
I20260807 11:11:19.822003 30927 leader_election.cc:288] T b8330847089e41b0811e28dc30529234 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:11:19.563151 25750 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 114836us
I20260807 11:11:19.574560 26036 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:11:19.825412 31034 raft_consensus.cc:2726] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 11:11:19.565340 31033 raft_consensus.cc:482] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:19.826290 31033 raft_consensus.cc:504] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:19.766924 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:19.828784 26039 raft_consensus.cc:2454] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:11:19.829051 30794 consensus_queue.cc:1046] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 11:11:19.830359 30794 consensus_queue.cc:1225] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer 4fd6381544574ddca559ce63b9243efc log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260807 11:11:19.830462 26033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 583) took 1892 ms. Trace:
I20260807 11:11:19.572086 30278 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:19.640046 25920 raft_consensus.cc:1226] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Rejecting Update request from peer b22d69db45194908ad3d345eba9badd9 for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:19.831416 30278 raft_consensus.cc:504] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:19.832348 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:19.613417 30475 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:19.613484 30749 raft_consensus.cc:482] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:19.648937 30717 raft_consensus.cc:482] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:19.649775 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5adbe0bf4741eb8b116a9bcf1cfc15" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:19.650462 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a291aa09434b06adbb86586dacec0e" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:11:19.833374 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 69633.5us
I20260807 11:11:19.833509 30475 raft_consensus.cc:504] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:19.834148 30278 leader_election.cc:288] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:19.834555 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:19.574546 31030 raft_consensus.cc:482] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:19.835634 25921 raft_consensus.cc:2379] T 674c5d3caeaf4ccabb23397e655b0e0a P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 8: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:11:19.653437 30171 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:19.615850 30955 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:19.836113 31030 raft_consensus.cc:504] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:19.837229 30955 raft_consensus.cc:504] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:19.837635 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:19.838809 30749 raft_consensus.cc:504] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:19.840099 30795 raft_consensus.cc:3032] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 8 LEADER]: Stepping down as leader of term 8
W20260807 11:11:19.654331 31017 log.cc:924] Time spent T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.378s user 0.003s sys 0.001s
I20260807 11:11:19.697670 30708 leader_election.cc:288] T bb5adbe0bf4741eb8b116a9bcf1cfc15 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:19.831044 30794 consensus_queue.cc:1046] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20260807 11:11:19.841250 30749 leader_election.cc:288] T 737af11139a541029b9473a8bfdb5980 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:19.841446 30717 raft_consensus.cc:504] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:19.841899 30794 raft_consensus.cc:3032] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 11:11:19.716750 25926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1883) took 2709 ms. Trace:
I20260807 11:11:19.842465 30794 raft_consensus.cc:729] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:19.843875 30717 leader_election.cc:288] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:19.842671 25926 rpcz_store.cc:276] 0807 11:11:16.976669 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:17.743089 (+766420us) service_pool.cc:227] Handling call
0807 11:11:17.743428 (+ 339us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:17.743688 (+ 260us) raft_consensus.cc:1520] Early marking committed up to index 5
0807 11:11:17.744013 (+ 325us) log.cc:842] Serialized 14 byte log entry
0807 11:11:19.593981 (+1849968us) log.cc:842] Serialized 14 byte log entry
0807 11:11:19.640651 (+ 46670us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:19.655325 (+ 14674us) raft_consensus.cc:1631] Marking committed up to 5
0807 11:11:19.665042 (+ 9717us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:19.684342 (+ 19300us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:19.686288 (+ 1946us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":463364,"prepare.queue_time_us":352627,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":79,"raft.queue_time_us":544915,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":4137,"thread_start_us":1303658,"threads_started":3,"wal-append.queue_time_us":951698}
I20260807 11:11:19.843719 30794 consensus_queue.cc:260] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:19.716727 25752 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:11:19.727387 30270 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
I20260807 11:11:19.729442 30741 raft_consensus.cc:2781] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 11:11:19.730302 26987 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50958 (request call id 531) took 2155 ms. Trace:
I20260807 11:11:19.748433 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35276 (request call id 406) took 2801 ms. Trace:
W20260807 11:11:19.752444 30925 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.013s sys 0.012s
W20260807 11:11:19.757797 25869 connection.cc:232] RPC call timeout handler was delayed by -1.1075580979231745s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21660000000000001s for a response
W20260807 11:11:19.513195 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 46364.3us
I20260807 11:11:19.773172 30256 raft_consensus.cc:504] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } attrs { replace: true } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:19.775072 30421 leader_election.cc:288] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 9 election: Requested vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:19.775234 30954 raft_consensus.cc:2726] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195) with higher term. Message: Invalid argument: T 38360ae0ac1547f9887cae7534bb515b P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 7. Current term is 8.
I20260807 11:11:19.769141 25925 rpcz_store.cc:276] 0807 11:11:16.851800 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:17.331384 (+479584us) service_pool.cc:227] Handling call
0807 11:11:17.360410 (+ 29026us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:17.360627 (+ 217us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 11:11:17.581490 (+220863us) log.cc:842] Serialized 14 byte log entry
0807 11:11:19.468935 (+1887445us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:19.506300 (+ 37365us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:19.511572 (+ 5272us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:19.539798 (+ 28226us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:19.539955 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1475848,"prepare.queue_time_us":283430,"prepare.run_cpu_time_us":2564,"prepare.run_wall_time_us":23564,"raft.queue_time_us":1576072,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":95,"thread_start_us":548865,"threads_started":3,"wal-append.queue_time_us":60331,"wal-append.run_cpu_time_us":6830,"wal-append.run_wall_time_us":1074892}
I20260807 11:11:19.770588 30922 raft_consensus.cc:686] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [term 7 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:11:19.781488 30949 raft_consensus.cc:3037] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:19.782115 26041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:11:19.786631 25981 leader_election.cc:334] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465): Timed out: RequestConsensusVote RPC to 127.23.120.130:36465 timed out after 2.104s (SENT)
I20260807 11:11:19.790196 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:19.851927 30794 raft_consensus.cc:3037] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:19.852682 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:19.853511 25916 raft_consensus.cc:2454] T 6ecbb49904f34f5aba0936699ba75cfb P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 5.
I20260807 11:11:19.854698 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:19.856003 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:19.845245 30955 leader_election.cc:288] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:19.845975 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a805d50b18af4d5eb99f7c96e790e1dd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:19.861166 25921 raft_consensus.cc:3037] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 4
I20260807 11:11:19.861490 25920 raft_consensus.cc:2454] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 5.
I20260807 11:11:19.863122 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:19.863219 30256 leader_election.cc:288] T f40bb008c149418989fdcd7c1e67d554 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:19.864392 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:19.864941 25981 leader_election.cc:302] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815
W20260807 11:11:19.866183 25981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 79590.3us
I20260807 11:11:19.865216 25917 rpcz_store.cc:276] 0807 11:11:16.947414 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:17.402717 (+455303us) service_pool.cc:227] Handling call
0807 11:11:17.415248 (+ 12531us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:19.446798 (+2031550us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:19.452228 (+ 5430us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:19.512141 (+ 59913us) log.cc:842] Serialized 26 byte log entry
0807 11:11:19.700420 (+188279us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:19.700464 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:19.702596 (+ 2132us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:19.748012 (+ 45416us) raft_consensus.cc:1671] finished
0807 11:11:19.748052 (+ 40us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:19.748210 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1631831,"raft.queue_time_us":1787423,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":172,"thread_start_us":482076,"threads_started":2,"wal-append.queue_time_us":189354}
I20260807 11:11:19.867198 30954 raft_consensus.cc:2726] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 11:11:19.867364 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:19.869316 25872 leader_election.cc:302] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:19.870532 30421 raft_consensus.cc:2781] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:19.870965 30421 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:19.871454 30421 raft_consensus.cc:3037] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:19.856603 25916 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 6.
I20260807 11:11:19.874192 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:19.874842 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8330847089e41b0811e28dc30529234" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
W20260807 11:11:19.876103 30925 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.666s user 0.013s sys 0.013s
I20260807 11:11:19.878108 30954 raft_consensus.cc:482] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:19.878716 26987 rpcz_store.cc:276] 0807 11:11:17.571783 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:17.571956 (+ 173us) service_pool.cc:227] Handling call
0807 11:11:17.587190 (+ 15234us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:19.278509 (+1691319us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:19.716893 (+438384us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b54004c13d4. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c4f51 00007f3fbf6c399e 00007f3fc46052e9 00007f3fbdceb684 00007f3fbdceb612 00007f3fbdceb541 00007f3fbdceb4bc 00007f3fbdcea772 00007f3fbdaee94c 00007f3fbdaedd86
0807 11:11:19.726727 (+ 9834us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3859378432}
I20260807 11:11:19.878878 30954 raft_consensus.cc:504] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:19.791978 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dbf7480b434c0ba6a902c817cf5455" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:19.792852 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:19.774503 26975 raft_consensus.cc:1226] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 3. Current term is 5. Ops: []
I20260807 11:11:19.878132 30922 consensus_queue.cc:237] T dee1fb79ff3944c08f589c435f2f09a9 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:19.883729 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:19.884479 25806 raft_consensus.cc:2454] T 8467fdc1886745c1a7049de84b30bcc0 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 7.
I20260807 11:11:19.796782 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:19.791134 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:19.792646 30981 raft_consensus.cc:504] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:19.885555 25799 raft_consensus.cc:2454] T a3d806783f4844d782ffd0c901a1dafb P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 5.
I20260807 11:11:19.885689 30954 leader_election.cc:288] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
W20260807 11:11:19.886628 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 245614us
I20260807 11:11:19.887073 30981 leader_election.cc:288] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:19.888038 30717 raft_consensus.cc:2781] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:19.888528 30717 raft_consensus.cc:482] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:19.840617 30795 raft_consensus.cc:729] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
I20260807 11:11:19.889003 30717 raft_consensus.cc:3037] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:19.889072 30708 consensus_queue.cc:1046] T a805d50b18af4d5eb99f7c96e790e1dd P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 11:11:19.889766 30795 consensus_queue.cc:260] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:19.814808 31023 raft_consensus.cc:504] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:19.892829 30741 raft_consensus.cc:482] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:19.892947 31023 leader_election.cc:288] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:19.893348 30741 raft_consensus.cc:3037] T 01cf60138c9d41bbb8d914b0a2736151 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:19.819428 26042 raft_consensus.cc:2454] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 8.
I20260807 11:11:19.894706 25919 raft_consensus.cc:2379] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 7: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:11:19.895419 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740366b034d646e09bc075abffba6286" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:19.567178 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:19.845678 31034 raft_consensus.cc:482] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:19.896677 25981 leader_election.cc:302] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:19.897576 30954 raft_consensus.cc:2781] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260807 11:11:19.809019 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 54701.4us
I20260807 11:11:19.836154 30475 leader_election.cc:288] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:19.719255 26046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43246 (request call id 573) took 1977 ms. Trace:
I20260807 11:11:19.898465 26046 rpcz_store.cc:276] 0807 11:11:17.742349 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:18.883665 (+1141316us) service_pool.cc:227] Handling call
0807 11:11:18.884226 (+ 561us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:18.908907 (+ 24681us) raft_consensus.cc:1520] Early marking committed up to index 7
0807 11:11:19.457521 (+548614us) log.cc:842] Serialized 14 byte log entry
0807 11:11:19.718707 (+261186us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:19.718885 (+ 178us) raft_consensus.cc:1631] Marking committed up to 7
0807 11:11:19.718964 (+ 79us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:19.719047 (+ 83us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:19.719192 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":71347,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"raft.queue_time_us":8554,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":2874,"thread_start_us":743752,"threads_started":2}
W20260807 11:11:19.878211 25869 connection.cc:232] RPC call timeout handler was delayed by -1.1075580979231745s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21660000000000001s for a response
W20260807 11:11:19.900240 25869 connection.cc:232] RPC call timeout handler was delayed by -1.0067580980248749s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2278s for a response
I20260807 11:11:19.900362 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e702bb04fc98457f9d8ee704fcaeafcf" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:19.896385 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:19.901198 25800 raft_consensus.cc:3037] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Advancing to term 10
I20260807 11:11:19.901372 25919 raft_consensus.cc:2454] T cccdf448d32448deae255de6bf7b4125 P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 5.
I20260807 11:11:19.902444 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:19.903062 25919 raft_consensus.cc:2454] T e4de8b347f214a69bbac09d43ad38db7 P 4fd6381544574ddca559ce63b9243efc [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 10.
I20260807 11:11:19.561782 30952 consensus_queue.cc:260] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:19.905146 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:19.905705 30256 consensus_queue.cc:1046] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 11:11:19.907032 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:19.907971 26042 raft_consensus.cc:2454] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 6.
I20260807 11:11:19.908169 31031 raft_consensus.cc:3032] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 3 LEADER]: Stepping down as leader of term 3
I20260807 11:11:19.908694 31031 raft_consensus.cc:729] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:19.903910 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:19.910112 26035 raft_consensus.cc:2379] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 7: Already voted for candidate b22d69db45194908ad3d345eba9badd9 in this term.
I20260807 11:11:19.909904 31031 consensus_queue.cc:260] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } attrs { replace: true } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:19.911589 31031 raft_consensus.cc:3037] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 3 FOLLOWER]: Advancing to term 5
I20260807 11:11:19.912237 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:19.912912 25796 raft_consensus.cc:2454] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 7.
I20260807 11:11:19.913767 25800 raft_consensus.cc:2454] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 10.
I20260807 11:11:19.905979 25799 raft_consensus.cc:2454] T f35a17e1683e41f4b146284f5cf09c58 P 606d49e053a346e486f48cb31832b815 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 10.
I20260807 11:11:19.913569 30724 consensus_queue.cc:1046] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 11:11:19.831043 26033 rpcz_store.cc:276] 0807 11:11:17.937821 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:19.066069 (+1128248us) service_pool.cc:227] Handling call
0807 11:11:19.097783 (+ 31714us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:19.103110 (+ 5327us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:19.103159 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:19.103901 (+ 742us) log.cc:842] Serialized 26 byte log entry
0807 11:11:19.383820 (+279919us) raft_consensus.cc:1631] Marking committed up to 6
0807 11:11:19.383855 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:19.383932 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:19.770009 (+386077us) spinlock_profiling.cc:243] Waited 6.44 ms on lock 0x7b24002c0378. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbc5e0f11 00007f3fbc5f4cf4 00007f3fbc5f2dfb 00007f3fbc681559 00007f3fbc6811e5 00007f3fbc68082d 00007f3fbf32f46e 00007f3fbf346c0c 00007f3fbf345f36 00007f3fbf345b58 00007f3fbf34f551
0807 11:11:19.830100 (+ 60091us) raft_consensus.cc:1671] finished
0807 11:11:19.830127 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:19.830397 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4450,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":147,"spinlock_wait_cycles":14817152,"thread_start_us":279446,"threads_started":1,"wal-append.queue_time_us":494993}
I20260807 11:11:19.914861 30749 consensus_queue.cc:1046] T 437f2acbd46c43b2834bf00f08d22f65 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260807 11:11:19.916422 29847 consensus_queue.cc:1046] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 11:11:19.917390 31045 consensus_queue.cc:1035] T e702bb04fc98457f9d8ee704fcaeafcf P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260807 11:11:19.918056 25869 leader_election.cc:302] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:19.940313 26938 leader_election.cc:302] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:19.898002 30954 raft_consensus.cc:482] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:19.897324 31034 raft_consensus.cc:504] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:19.922142 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:19.925130 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:19.932010 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:19.938732 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:19.946645 25798 raft_consensus.cc:3037] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 3 FOLLOWER]: Advancing to term 5
I20260807 11:11:19.919852 30717 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:19.946972 25806 raft_consensus.cc:3037] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:19.937793 25981 leader_election.cc:302] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:19.947741 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:19.949884 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:19.951072 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:19.947685 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8467fdc1886745c1a7049de84b30bcc0" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:19.953876 25981 leader_election.cc:302] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
W20260807 11:11:19.956969 25872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 87799.8us
I20260807 11:11:19.951546 25921 raft_consensus.cc:2454] T a805d50b18af4d5eb99f7c96e790e1dd P 4fd6381544574ddca559ce63b9243efc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 4.
I20260807 11:11:19.838843 31030 leader_election.cc:288] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:19.964633 30724 consensus_queue.cc:1225] T d8012276091e4b63b062cb7152208a16 P eb1fca62dcb34ed9b32188ccb19f0543 [NON_LEADER]: Peer 606d49e053a346e486f48cb31832b815 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260807 11:11:19.965515 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:19.966372 26975 raft_consensus.cc:3037] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Advancing to term 8
I20260807 11:11:19.966434 25920 raft_consensus.cc:2454] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 10.
I20260807 11:11:19.967432 25799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:19.967708 31034 leader_election.cc:288] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:19.968436 30884 consensus_queue.cc:1046] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 11:11:19.969821 25926 raft_consensus.cc:2379] T 97a61636680c4b4a805a504dfcb4916f P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 8: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:11:19.970728 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955db4a12dc04bfe8124047fc46823d3" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:19.970831 31033 leader_election.cc:288] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:19.971890 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:19.973312 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacbdb7bffca4346a1e00e7b1e90fdea" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:19.972699 25926 raft_consensus.cc:2379] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 in current term 7: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:11:19.978161 25919 raft_consensus.cc:2454] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:11:19.982218 25801 raft_consensus.cc:1226] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:19.985966 25981 leader_election.cc:302] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:19.951462 30954 raft_consensus.cc:3037] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
W20260807 11:11:19.992743 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 74775.6us
I20260807 11:11:19.994524 30852 raft_consensus.cc:2726] T 66d10b6b12404f0488b9f83c8080069f P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:19.994323 25869 leader_election.cc:302] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:19.949394 29847 raft_consensus.cc:3032] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Stepping down as leader of term 6
I20260807 11:11:19.995680 25919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:19.997778 29847 raft_consensus.cc:729] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:20.002275 29847 consensus_queue.cc:260] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:20.003996 29847 raft_consensus.cc:3037] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:20.006398 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:20.007364 25925 raft_consensus.cc:2379] T d5d5c9aff2844213a67d710585224083 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate eb1fca62dcb34ed9b32188ccb19f0543 in current term 9: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:11:20.008279 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:20.009657 25919 raft_consensus.cc:3037] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:20.011977 25925 raft_consensus.cc:2454] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 7.
I20260807 11:11:20.013031 30959 raft_consensus.cc:2781] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 11:11:20.013566 30959 raft_consensus.cc:482] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:20.013872 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb81660c5a8b4365a928d2ef1e4d0be5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:20.014117 30959 raft_consensus.cc:3037] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:20.016177 25981 leader_election.cc:302] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:20.016891 25932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1885) took 2892 ms. Trace:
I20260807 11:11:20.017431 25981 leader_election.cc:302] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:20.018471 30977 raft_consensus.cc:2726] T f40bb008c149418989fdcd7c1e67d554 P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 11:11:20.017279 25932 rpcz_store.cc:276] 0807 11:11:17.124473 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:18.518623 (+1394150us) service_pool.cc:227] Handling call
0807 11:11:18.553160 (+ 34537us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 11:11:19.566906 (+1013746us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:19.569272 (+ 2366us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 11:11:19.569758 (+ 486us) log.cc:842] Serialized 26 byte log entry
0807 11:11:19.705311 (+135553us) raft_consensus.cc:1631] Marking committed up to 3
0807 11:11:19.705364 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:19.716279 (+ 10915us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:20.016531 (+300252us) raft_consensus.cc:1671] finished
0807 11:11:20.016550 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:20.016692 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":641008,"raft.queue_time_us":651350,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"thread_start_us":437063,"threads_started":2,"wal-append.queue_time_us":444710}
I20260807 11:11:20.018656 25981 leader_election.cc:302] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:20.018949 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:20.019562 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee1fb79ff3944c08f589c435f2f09a9" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:20.019632 25921 raft_consensus.cc:2454] T 2f60eb5499d64c89b971928b13bcd1be P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 7.
I20260807 11:11:20.020262 25932 raft_consensus.cc:2362] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 7. Re-sending same reply.
I20260807 11:11:20.020740 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0766ccf18b84e41ac68eb6599a834c5" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:20.021226 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:20.020545 25981 leader_election.cc:398] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 election: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 7. Current term is 8.
I20260807 11:11:19.982570 30475 raft_consensus.cc:2781] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 11:11:20.021865 25932 raft_consensus.cc:2454] T f9465d2caa3e43c8891f52350177df8d P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 7.
I20260807 11:11:20.022051 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff0e05310b145358043bba0a9a8ba11" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:19.999987 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737af11139a541029b9473a8bfdb5980" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc"
I20260807 11:11:20.021518 25921 raft_consensus.cc:2454] T e0766ccf18b84e41ac68eb6599a834c5 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 7.
I20260807 11:11:20.022136 25981 leader_election.cc:401] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260807 11:11:19.932948 30421 raft_consensus.cc:504] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:19.985119 30927 raft_consensus.cc:2781] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 11:11:19.995757 30955 raft_consensus.cc:2781] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 11:11:20.017285 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:11:20.044021 26938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 44406.7us
I20260807 11:11:20.020274 29991 raft_consensus.cc:2726] T 437f2acbd46c43b2834bf00f08d22f65 P b22d69db45194908ad3d345eba9badd9 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260807 11:11:20.036892 30977 consensus_queue.cc:1035] T b7c44ad3ac214b2d83c41bc5aa01f349 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20260807 11:11:20.042276 30997 raft_consensus.cc:2726] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279) with higher term. Message: Invalid argument: T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for earlier term 7. Current term is 8.
I20260807 11:11:20.018994 30987 raft_consensus.cc:2726] T cf86a21774c34f33a6540d1dad0de0c4 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 11:11:20.051951 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:20.047851 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:20.052693 25921 raft_consensus.cc:2454] T cf86a21774c34f33a6540d1dad0de0c4 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 6.
I20260807 11:11:20.053572 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:20.054440 25932 raft_consensus.cc:2454] T d47975fbb6e3412aaf146ed0794f986c P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 6.
I20260807 11:11:20.053633 25921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be451f12275472a9c2c53877d9d1a0a" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:20.018677 30717 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 election: Requested vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:20.025070 30475 raft_consensus.cc:482] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:20.068540 25916 raft_consensus.cc:2379] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 in current term 7: Already voted for candidate 4fd6381544574ddca559ce63b9243efc in this term.
I20260807 11:11:20.068681 25917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998ef34204b64d2481698cc29e055cc2" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:20.066565 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cf60138c9d41bbb8d914b0a2736151" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:20.045181 30955 raft_consensus.cc:482] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:20.065357 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:20.041759 30927 raft_consensus.cc:482] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4fd6381544574ddca559ce63b9243efc)
I20260807 11:11:20.070776 25932 raft_consensus.cc:2454] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 6.
I20260807 11:11:20.041522 31045 raft_consensus.cc:2726] T 9c6408e9acee46adacd2bbd81669f27a P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260807 11:11:20.074993 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 80951.3us
I20260807 11:11:20.075314 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:20.082067 25920 raft_consensus.cc:2454] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 7.
I20260807 11:11:20.083259 26034 raft_consensus.cc:1261] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260807 11:11:20.083516 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:20.070381 30475 raft_consensus.cc:3037] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:20.092870 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3395e44a980a4ac2bc07304ae2525b35" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:20.076314 30955 raft_consensus.cc:3037] T 0e15f851c3b34d9893d8a5c89e0f98ff P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:20.095906 25925 raft_consensus.cc:2396] T 3c69ebea3aeb47d48ab8e083fdf71f7c P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b22d69db45194908ad3d345eba9badd9 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260807 11:11:20.096635 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d531084c9447b78cffb286d6b966b7" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:20.080749 30927 raft_consensus.cc:3037] T a3d806783f4844d782ffd0c901a1dafb P b22d69db45194908ad3d345eba9badd9 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:20.128715 25920 raft_consensus.cc:2396] T 93d531084c9447b78cffb286d6b966b7 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 606d49e053a346e486f48cb31832b815 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260807 11:11:20.135812 25869 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:11:20.139954 25932 raft_consensus.cc:1226] T 0fc5b1dfff1a4556abdd562afbebd768 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 8. Current term is 9. Ops: []
W20260807 11:11:20.144317 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 51225us
I20260807 11:11:20.144503 25925 raft_consensus.cc:1226] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 8. Current term is 9. Ops: []
I20260807 11:11:20.144465 25926 raft_consensus.cc:1261] T b7c44ad3ac214b2d83c41bc5aa01f349 P 4fd6381544574ddca559ce63b9243efc [term 10 FOLLOWER]: Refusing update from remote peer b22d69db45194908ad3d345eba9badd9: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260807 11:11:20.178387 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe28118d3d4770aa8de6f8d38ca61f" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
W20260807 11:11:20.184892 25752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.131:46009, user_credentials={real_user=slave}} blocked reactor thread for 296233us
I20260807 11:11:20.185928 25932 raft_consensus.cc:1226] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Rejecting Update request from peer eb1fca62dcb34ed9b32188ccb19f0543 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260807 11:11:20.098904 30171 consensus_queue.cc:1035] T 2f60eb5499d64c89b971928b13bcd1be P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.075s
I20260807 11:11:20.173856 30852 raft_consensus.cc:2781] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader election won for term 7
I20260807 11:11:20.200517 25926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d806783f4844d782ffd0c901a1dafb" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:20.203186 25921 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P 4fd6381544574ddca559ce63b9243efc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:11:20.195886 25808 raft_consensus.cc:2454] T 97a61636680c4b4a805a504dfcb4916f P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 8.
I20260807 11:11:20.223852 25798 raft_consensus.cc:2454] T 7be451f12275472a9c2c53877d9d1a0a P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 5.
I20260807 11:11:20.231815 25926 raft_consensus.cc:1261] T f75da3017fd744719647332808daba49 P 4fd6381544574ddca559ce63b9243efc [term 9 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260807 11:11:20.233552 25916 raft_consensus.cc:1261] T eb34326013f64140ac84b4fdaff38542 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Refusing update from remote peer 606d49e053a346e486f48cb31832b815: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260807 11:11:20.238893 25869 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 54322.7us
I20260807 11:11:20.258702 30795 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Advancing to term 9
I20260807 11:11:20.261734 30717 raft_consensus.cc:482] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb1fca62dcb34ed9b32188ccb19f0543)
I20260807 11:11:20.262450 30717 raft_consensus.cc:504] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:20.266458 25981 leader_election.cc:302] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:20.266690 25920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cb005020884d87b17995a28b8effd6" candidate_uuid: "606d49e053a346e486f48cb31832b815" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4fd6381544574ddca559ce63b9243efc" is_pre_election: true
I20260807 11:11:20.302692 30717 leader_election.cc:288] T f31f65eddcdf48f6a64e0a1ddc0c6695 P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
W20260807 11:11:20.303779 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.130:36465, user_credentials={real_user=slave}} blocked reactor thread for 331259us
W20260807 11:11:20.323984 31066 log.cc:924] Time spent T 737af11139a541029b9473a8bfdb5980 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20260807 11:11:20.330194 25869 leader_election.cc:302] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
W20260807 11:11:20.333038 31063 log.cc:924] Time spent T adab2949574e4c6badfcf68e51b9c072 P b22d69db45194908ad3d345eba9badd9: Append to log took a long time: real 0.104s user 0.000s sys 0.001s
I20260807 11:11:20.338666 26975 raft_consensus.cc:2454] T 998ef34204b64d2481698cc29e055cc2 P eb1fca62dcb34ed9b32188ccb19f0543 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 8.
I20260807 11:11:20.267423 30852 raft_consensus.cc:686] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [term 7 LEADER]: Becoming Leader. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:20.349720 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:20.351804 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:20.303740 30977 raft_consensus.cc:2781] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 11:11:20.312497 30987 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:20.356827 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:20.361357 26980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:11:20.358505 25639 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 66d10b6b12404f0488b9f83c8080069f with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica b22d69db45194908ad3d345eba9badd9 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 leader_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } attrs { replace: true } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } } }
I20260807 11:11:20.310875 30421 leader_election.cc:288] T 85d06c4d618c4aa6af896f216680dd68 P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 7 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
I20260807 11:11:20.364037 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4de8b347f214a69bbac09d43ad38db7" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:20.365108 25801 raft_consensus.cc:2396] T e4de8b347f214a69bbac09d43ad38db7 P 606d49e053a346e486f48cb31832b815 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260807 11:11:20.369544 26975 raft_consensus.cc:2396] T e4de8b347f214a69bbac09d43ad38db7 P eb1fca62dcb34ed9b32188ccb19f0543 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260807 11:11:20.372860 26985 raft_consensus.cc:1226] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 10 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 9. Current term is 10. Ops: []
I20260807 11:11:20.297703 31031 consensus_queue.cc:1046] T 740366b034d646e09bc075abffba6286 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260807 11:11:20.376583 26034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:20.352850 30270 raft_consensus.cc:2726] T 51fb619c2ada475f817fbcb9739a401a P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 11:11:20.380137 26035 raft_consensus.cc:1261] T 98fe28118d3d4770aa8de6f8d38ca61f P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 11:11:20.306083 30959 raft_consensus.cc:504] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:20.395041 25919 raft_consensus.cc:2454] T 38360ae0ac1547f9887cae7534bb515b P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606d49e053a346e486f48cb31832b815 in term 8.
I20260807 11:11:20.373041 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:20.356129 30977 raft_consensus.cc:482] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:20.398411 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35a17e1683e41f4b146284f5cf09c58" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:20.402870 26975 raft_consensus.cc:2362] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 10. Re-sending same reply.
I20260807 11:11:20.406023 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d10b6b12404f0488b9f83c8080069f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:20.352259 30256 consensus_queue.cc:1046] T b8330847089e41b0811e28dc30529234 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20260807 11:11:20.423334 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38360ae0ac1547f9887cae7534bb515b" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:20.423877 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c69ebea3aeb47d48ab8e083fdf71f7c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:20.414819 25803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:20.414953 30852 consensus_queue.cc:237] T 84f78f4141b342d9b7e397096f34af59 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:20.415663 25786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:20.416610 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9465d2caa3e43c8891f52350177df8d" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:20.418340 25806 raft_consensus.cc:2454] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 7.
I20260807 11:11:20.429394 25981 leader_election.cc:302] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:20.427743 25786 raft_consensus.cc:2454] T f5e378b8a19447d69c6c9433f071086e P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 8.
I20260807 11:11:20.425136 25803 raft_consensus.cc:2454] T 0e15f851c3b34d9893d8a5c89e0f98ff P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 7.
I20260807 11:11:20.429339 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543"
I20260807 11:11:20.430415 25796 raft_consensus.cc:1226] T 3395e44a980a4ac2bc07304ae2525b35 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:20.427045 26033 raft_consensus.cc:2379] T 38360ae0ac1547f9887cae7534bb515b P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 8: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:11:20.426301 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:20.405352 30977 raft_consensus.cc:3037] T 97a61636680c4b4a805a504dfcb4916f P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:20.366622 30987 raft_consensus.cc:504] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:20.371405 30278 consensus_queue.cc:1046] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260807 11:11:20.435390 29991 raft_consensus.cc:2781] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 11:11:20.372983 31033 consensus_queue.cc:1035] T 98fe28118d3d4770aa8de6f8d38ca61f P 4fd6381544574ddca559ce63b9243efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260807 11:11:20.437775 29991 raft_consensus.cc:686] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [term 7 LEADER]: Becoming Leader. State: Replica: b22d69db45194908ad3d345eba9badd9, State: Running, Role: LEADER
I20260807 11:11:20.444572 26037 raft_consensus.cc:1261] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 9 FOLLOWER]: Refusing update from remote peer eb1fca62dcb34ed9b32188ccb19f0543: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260807 11:11:20.447422 26038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfca1b0bf4c479a834433a4c0ed66ac" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:20.448204 26038 raft_consensus.cc:3037] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Advancing to term 9
I20260807 11:11:20.451020 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:11:20.445431 25931 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.002s sys 0.022s
W20260807 11:11:20.453029 25931 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.727s user 0.003s sys 0.022s
I20260807 11:11:20.454655 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75da3017fd744719647332808daba49" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:20.451869 26976 raft_consensus.cc:2454] T f75da3017fd744719647332808daba49 P eb1fca62dcb34ed9b32188ccb19f0543 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 9.
I20260807 11:11:20.457455 26034 raft_consensus.cc:2396] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 2.
I20260807 11:11:20.437925 26974 raft_consensus.cc:1226] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 FOLLOWER]: Rejecting Update request from peer 606d49e053a346e486f48cb31832b815 for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:20.459404 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5d3caeaf4ccabb23397e655b0e0a" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:20.458855 30918 consensus_queue.cc:1035] T d5d5c9aff2844213a67d710585224083 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
I20260807 11:11:20.465014 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:20.467197 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:20.475183 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61636680c4b4a805a504dfcb4916f" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:20.461645 30695 consensus_queue.cc:1035] T f40bb008c149418989fdcd7c1e67d554 P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.013s
I20260807 11:11:20.520473 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
W20260807 11:11:20.519315 25918 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.707s user 0.017s sys 0.012s
W20260807 11:11:20.522291 25918 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.804s user 0.018s sys 0.013s
W20260807 11:11:20.469475 30884 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.500s user 0.045s sys 0.073s
I20260807 11:11:20.558204 26975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8012276091e4b63b062cb7152208a16" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:20.467217 30278 consensus_queue.cc:1225] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [LEADER]: Peer 606d49e053a346e486f48cb31832b815 log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260807 11:11:20.544368 25931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1882) took 3684 ms. Trace:
I20260807 11:11:20.445734 30954 raft_consensus.cc:504] T 6ecbb49904f34f5aba0936699ba75cfb P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:20.561452 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:20.563536 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d06c4d618c4aa6af896f216680dd68" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:20.474251 30997 raft_consensus.cc:482] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:20.578368 26985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e5feeda1eb4affb2532bda1b0055cd" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:20.580729 25918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39742 (request call id 1881) took 3721 ms. Trace:
I20260807 11:11:20.584203 25808 raft_consensus.cc:2379] T 85d06c4d618c4aa6af896f216680dd68 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 7: Already voted for candidate 606d49e053a346e486f48cb31832b815 in this term.
I20260807 11:11:20.585700 25801 raft_consensus.cc:1226] T e702bb04fc98457f9d8ee704fcaeafcf P 606d49e053a346e486f48cb31832b815 [term 10 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260807 11:11:20.610029 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" is_pre_election: true
I20260807 11:11:20.581282 26985 raft_consensus.cc:2454] T 99e5feeda1eb4affb2532bda1b0055cd P eb1fca62dcb34ed9b32188ccb19f0543 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 6.
I20260807 11:11:20.612941 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34326013f64140ac84b4fdaff38542" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:11:20.561378 30884 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.589s user 0.045s sys 0.073s
I20260807 11:11:20.473628 30681 consensus_queue.cc:1046] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.089s
I20260807 11:11:20.626997 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e15f851c3b34d9893d8a5c89e0f98ff" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:20.628420 26039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d5c9aff2844213a67d710585224083" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:20.627884 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdf448d32448deae255de6bf7b4125" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:20.623859 26033 raft_consensus.cc:2454] T eb34326013f64140ac84b4fdaff38542 P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:11:20.643695 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf86a21774c34f33a6540d1dad0de0c4" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:20.450687 29991 consensus_queue.cc:237] T 3395e44a980a4ac2bc07304ae2525b35 P b22d69db45194908ad3d345eba9badd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:20.645931 26035 raft_consensus.cc:1226] T 61cb005020884d87b17995a28b8effd6 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:20.565148 25931 rpcz_store.cc:276] 0807 11:11:16.859654 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:17.384022 (+524368us) service_pool.cc:227] Handling call
0807 11:11:17.398744 (+ 14722us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 11:11:17.399024 (+ 280us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 11:11:17.592931 (+193907us) log.cc:842] Serialized 14 byte log entry
0807 11:11:19.704663 (+2111732us) log.cc:842] Serialized 14 byte log entry
0807 11:11:20.463994 (+759331us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 11:11:20.468845 (+ 4851us) raft_consensus.cc:1631] Marking committed up to 4
0807 11:11:20.472326 (+ 3481us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:20.544136 (+ 71810us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:20.544295 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1428535,"prepare.queue_time_us":475403,"prepare.run_cpu_time_us":133,"prepare.run_wall_time_us":3314,"raft.queue_time_us":1501413,"raft.run_cpu_time_us":375,"raft.run_wall_time_us":5055,"thread_start_us":1421673,"threads_started":5,"wal-append.queue_time_us":364655,"wal-append.run_cpu_time_us":1848,"wal-append.run_wall_time_us":1167300}
W20260807 11:11:20.616387 31056 log.cc:924] Time spent T cb81660c5a8b4365a928d2ef1e4d0be5 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260807 11:11:20.655028 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc5b1dfff1a4556abdd562afbebd768" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
W20260807 11:11:20.657599 25750 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.736s user 0.020s sys 0.037s
W20260807 11:11:20.658862 25750 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.737s user 0.020s sys 0.037s
W20260807 11:11:20.615963 30708 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.726s user 0.031s sys 0.065s
I20260807 11:11:20.582330 25918 rpcz_store.cc:276] 0807 11:11:16.859329 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:17.371624 (+512295us) service_pool.cc:227] Handling call
0807 11:11:17.393817 (+ 22193us) raft_consensus.cc:1457] Updating replica for 3 ops
0807 11:11:19.470076 (+2076259us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 11:11:19.477008 (+ 6932us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0807 11:11:19.477860 (+ 852us) log.cc:842] Serialized 78 byte log entry
0807 11:11:19.709810 (+231950us) raft_consensus.cc:1631] Marking committed up to 2
0807 11:11:20.522738 (+812928us) log.cc:842] Serialized 14 byte log entry
0807 11:11:20.527324 (+ 4586us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 11:11:20.559456 (+ 32132us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 11:11:20.559505 (+ 49us) raft_consensus.cc:1671] finished
0807 11:11:20.559517 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 11:11:20.580657 (+ 21140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1660253,"prepare.queue_time_us":18060,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":1813328,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":14955,"thread_start_us":1305027,"threads_started":3,"wal-append.queue_time_us":320969}
W20260807 11:11:20.646330 29847 consensus_meta.cc:287] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc: Time spent flushing consensus metadata: real 0.642s user 0.008s sys 0.004s
W20260807 11:11:20.629997 30922 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.743s user 0.028s sys 0.067s
I20260807 11:11:20.677011 25917 raft_consensus.cc:2454] T 998ef34204b64d2481698cc29e055cc2 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1fca62dcb34ed9b32188ccb19f0543 in term 7.
W20260807 11:11:20.661521 30650 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.803s user 0.051s sys 0.099s
I20260807 11:11:20.681583 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c44ad3ac214b2d83c41bc5aa01f349" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:11:20.663560 30708 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.773s user 0.031s sys 0.065s
I20260807 11:11:20.684957 25808 raft_consensus.cc:1226] T dacbdb7bffca4346a1e00e7b1e90fdea P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 4. Current term is 5. Ops: []
I20260807 11:11:20.685772 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47975fbb6e3412aaf146ed0794f986c" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:11:20.687633 25797 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.763s user 0.020s sys 0.048s
W20260807 11:11:20.687958 25797 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.763s user 0.020s sys 0.048s
I20260807 11:11:20.657428 31034 raft_consensus.cc:482] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:20.686586 26042 raft_consensus.cc:2454] T d47975fbb6e3412aaf146ed0794f986c P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 6.
I20260807 11:11:20.693343 25869 leader_election.cc:302] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fd6381544574ddca559ce63b9243efc, 606d49e053a346e486f48cb31832b815; no voters:
I20260807 11:11:20.689651 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f60eb5499d64c89b971928b13bcd1be" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:20.698017 26035 raft_consensus.cc:2379] T 2f60eb5499d64c89b971928b13bcd1be P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 8: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
W20260807 11:11:20.674903 30922 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.788s user 0.029s sys 0.068s
I20260807 11:11:20.618431 30997 raft_consensus.cc:504] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:20.700814 25798 raft_consensus.cc:1226] T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Rejecting Update request from peer 4fd6381544574ddca559ce63b9243efc for earlier term 6. Current term is 7. Ops: []
I20260807 11:11:20.704600 26936 leader_election.cc:302] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:11:20.688575 30650 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.825s user 0.052s sys 0.100s
I20260807 11:11:20.731777 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bb008c149418989fdcd7c1e67d554" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
W20260807 11:11:20.698026 30949 consensus_meta.cc:287] T 6eed5249afac4e16a4b17938ad6e1353 P eb1fca62dcb34ed9b32188ccb19f0543: Time spent flushing consensus metadata: real 0.829s user 0.013s sys 0.000s
I20260807 11:11:20.674201 31033 raft_consensus.cc:482] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
W20260807 11:11:20.682708 30925 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.806s user 0.028s sys 0.079s
I20260807 11:11:20.733971 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e378b8a19447d69c6c9433f071086e" candidate_uuid: "4fd6381544574ddca559ce63b9243efc" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9" is_pre_election: true
I20260807 11:11:20.743681 26046 raft_consensus.cc:2454] T f5e378b8a19447d69c6c9433f071086e P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd6381544574ddca559ce63b9243efc in term 8.
I20260807 11:11:20.726943 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f78f4141b342d9b7e397096f34af59" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:20.744726 26039 raft_consensus.cc:2379] T d5d5c9aff2844213a67d710585224083 P b22d69db45194908ad3d345eba9badd9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc in current term 9: Already voted for candidate eb1fca62dcb34ed9b32188ccb19f0543 in this term.
I20260807 11:11:20.689602 30884 raft_consensus.cc:3032] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 11:11:20.751156 26936 leader_election.cc:302] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
I20260807 11:11:20.738883 26033 raft_consensus.cc:1261] T 737af11139a541029b9473a8bfdb5980 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Refusing update from remote peer 4fd6381544574ddca559ce63b9243efc: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 11:11:20.732739 30681 raft_consensus.cc:3032] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 LEADER]: Stepping down as leader of term 7
W20260807 11:11:20.743858 30749 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.780s user 0.040s sys 0.049s
W20260807 11:11:20.750535 30925 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.874s user 0.028s sys 0.080s
W20260807 11:11:20.744016 31053 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.749s user 0.038s sys 0.060s
W20260807 11:11:20.827571 31053 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.832s user 0.038s sys 0.060s
W20260807 11:11:20.832834 25986 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.853s user 0.018s sys 0.035s
W20260807 11:11:20.833247 25986 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.042s user 0.019s sys 0.035s
I20260807 11:11:20.833940 30922 raft_consensus.cc:482] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 11:11:20.834753 30795 consensus_meta.cc:287] T cbfca1b0bf4c479a834433a4c0ed66ac P 606d49e053a346e486f48cb31832b815: Time spent flushing consensus metadata: real 0.575s user 0.007s sys 0.006s
I20260807 11:11:20.835032 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b22d69db45194908ad3d345eba9badd9"
I20260807 11:11:20.835700 26033 raft_consensus.cc:3037] T b1765495856a47f09ddbb303f76c635f P b22d69db45194908ad3d345eba9badd9 [term 6 FOLLOWER]: Advancing to term 7
W20260807 11:11:20.823055 26936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.120.129:40195, user_credentials={real_user=slave}} blocked reactor thread for 79411.6us
I20260807 11:11:20.817531 30270 raft_consensus.cc:2781] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260807 11:11:20.818264 30749 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.869s user 0.040s sys 0.049s
I20260807 11:11:20.834436 30922 raft_consensus.cc:504] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:20.840905 25797 raft_consensus.cc:2454] T 84f78f4141b342d9b7e397096f34af59 P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 7.
I20260807 11:11:20.799544 30959 leader_election.cc:288] T b1765495856a47f09ddbb303f76c635f P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 7 election: Requested vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), b22d69db45194908ad3d345eba9badd9 (127.23.120.131:46009)
W20260807 11:11:20.852937 30882 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.685s user 0.031s sys 0.055s
I20260807 11:11:20.750976 30884 raft_consensus.cc:729] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 606d49e053a346e486f48cb31832b815, State: Running, Role: LEADER
W20260807 11:11:20.853294 30882 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.798s user 0.031s sys 0.055s
W20260807 11:11:20.854094 30985 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.862s user 0.023s sys 0.031s
W20260807 11:11:20.854537 30985 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.060s user 0.024s sys 0.031s
I20260807 11:11:20.854050 30884 consensus_queue.cc:260] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:20.854924 30985 raft_consensus.cc:482] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:20.854059 31088 raft_consensus.cc:482] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:11:20.855455 30985 raft_consensus.cc:504] T 986fe968b7be4eab9c480ec85c0384b0 P b22d69db45194908ad3d345eba9badd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:20.856148 31045 raft_consensus.cc:3032] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Stepping down as leader of term 6
I20260807 11:11:20.856659 31045 raft_consensus.cc:729] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4fd6381544574ddca559ce63b9243efc, State: Running, Role: LEADER
I20260807 11:11:20.856245 31088 raft_consensus.cc:504] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" member_type: VOTER last_known_addr { host: "127.23.120.132" port: 43279 } attrs { promote: false } }
I20260807 11:11:20.857359 25981 leader_election.cc:302] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9; no voters:
I20260807 11:11:20.857685 31045 consensus_queue.cc:260] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:20.858831 31088 leader_election.cc:288] T 51fb619c2ada475f817fbcb9739a401a P 606d49e053a346e486f48cb31832b815 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465), eb1fca62dcb34ed9b32188ccb19f0543 (127.23.120.132:43279)
I20260807 11:11:20.858999 31045 raft_consensus.cc:3037] T 61cb005020884d87b17995a28b8effd6 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Advancing to term 7
W20260807 11:11:20.863590 30475 consensus_meta.cc:287] T 8467fdc1886745c1a7049de84b30bcc0 P b22d69db45194908ad3d345eba9badd9: Time spent flushing consensus metadata: real 0.729s user 0.009s sys 0.000s
I20260807 11:11:20.848194 30925 raft_consensus.cc:2781] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 10 FOLLOWER]: Leader election won for term 10
I20260807 11:11:20.850505 30922 leader_election.cc:288] T 409fcb387861467396e84e2b22b8f9be P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:20.799180 30681 raft_consensus.cc:729] T 674c5d3caeaf4ccabb23397e655b0e0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:20.792547 30708 consensus_queue.cc:1035] T eb34326013f64140ac84b4fdaff38542 P 606d49e053a346e486f48cb31832b815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.111s
W20260807 11:11:20.797034 31057 log.cc:924] Time spent T 955db4a12dc04bfe8124047fc46823d3 P 606d49e053a346e486f48cb31832b815: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260807 11:11:20.869102 25917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35276 (request call id 420) took 1006 ms. Trace:
I20260807 11:11:20.879443 26936 leader_election.cc:302] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 606d49e053a346e486f48cb31832b815, b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters:
W20260807 11:11:20.824096 31067 log.cc:924] Time spent T 2ff0e05310b145358043bba0a9a8ba11 P 4fd6381544574ddca559ce63b9243efc: Append to log took a long time: real 0.093s user 0.001s sys 0.000s
I20260807 11:11:20.728924 31034 raft_consensus.cc:504] T dee1fb79ff3944c08f589c435f2f09a9 P 4fd6381544574ddca559ce63b9243efc [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
W20260807 11:11:20.826632 30924 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.765s user 0.018s sys 0.033s
I20260807 11:11:20.879971 25917 rpcz_store.cc:276] 0807 11:11:19.863133 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 11:11:20.047477 (+184344us) service_pool.cc:227] Handling call
0807 11:11:20.868874 (+821397us) spinlock_profiling.cc:243] Waited 607 ms on lock 0x7b5400408654. stack: 00007f3fbc76a033 00007f3fbc769f30 00007f3fbc0c8534 00007f3fc5dd48b4 00007f3fc5dd4849 00007f3fbf649471 00007f3fbf6c82ae 00007f3fc4605c27 00007f3fbdcedc04 00007f3fbdcedb92 00007f3fbdcedac1 00007f3fbdceda3c 00007f3fbdceccf2 00007f3fbdaee94c 00007f3fbdaedd86 00007f3fbdaed72f
0807 11:11:20.869002 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1397197056}
I20260807 11:11:20.846170 30630 raft_consensus.cc:2781] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260807 11:11:20.843390 30270 raft_consensus.cc:482] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:20.879527 25796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecbb49904f34f5aba0936699ba75cfb" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:20.923594 25796 raft_consensus.cc:3037] T 6ecbb49904f34f5aba0936699ba75cfb P 606d49e053a346e486f48cb31832b815 [term 5 FOLLOWER]: Advancing to term 6
I20260807 11:11:20.750828 31033 raft_consensus.cc:504] T a09a74fdae5d44c8a9b375e1cfdea1e9 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129" port: 40195 } } peers { permanent_uuid: "4fd6381544574ddca559ce63b9243efc" member_type: VOTER last_known_addr { host: "127.23.120.130" port: 36465 } } peers { permanent_uuid: "b22d69db45194908ad3d345eba9badd9" member_type: VOTER last_known_addr { host: "127.23.120.131" port: 46009 } }
I20260807 11:11:20.837503 30987 leader_election.cc:288] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 606d49e053a346e486f48cb31832b815 (127.23.120.129:40195), 4fd6381544574ddca559ce63b9243efc (127.23.120.130:36465)
I20260807 11:11:20.893234 26038 raft_consensus.cc:2396] T cbfca1b0bf4c479a834433a4c0ed66ac P b22d69db45194908ad3d345eba9badd9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fd6381544574ddca559ce63b9243efc for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260807 11:11:20.889963 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409fcb387861467396e84e2b22b8f9be" candidate_uuid: "b22d69db45194908ad3d345eba9badd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815" is_pre_election: true
I20260807 11:11:20.926784 25981 leader_election.cc:302] T d8012276091e4b63b062cb7152208a16 P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b22d69db45194908ad3d345eba9badd9; no voters: 606d49e053a346e486f48cb31832b815, eb1fca62dcb34ed9b32188ccb19f0543
I20260807 11:11:20.929016 25797 raft_consensus.cc:2454] T 409fcb387861467396e84e2b22b8f9be P 606d49e053a346e486f48cb31832b815 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22d69db45194908ad3d345eba9badd9 in term 8.
W20260807 11:11:20.917659 30924 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.059s user 0.018s sys 0.033s
I20260807 11:11:20.930987 31043 raft_consensus.cc:482] T 76dbf7480b434c0ba6a902c817cf5455 P 4fd6381544574ddca559ce63b9243efc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 606d49e053a346e486f48cb31832b815)
I20260807 11:11:20.886771 30922 raft_consensus.cc:2781] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 11:11:20.931953 25981 leader_election.cc:302] T 99e5feeda1eb4affb2532bda1b0055cd P b22d69db45194908ad3d345eba9badd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b22d69db45194908ad3d345eba9badd9, eb1fca62dcb34ed9b32188ccb19f0543; no voters: 606d49e053a346e486f48cb31832b815
I20260807 11:11:20.883126 30925 raft_consensus.cc:686] T e702bb04fc98457f9d8ee704fcaeafcf P eb1fca62dcb34ed9b32188ccb19f0543 [term 10 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:20.917567 30630 raft_consensus.cc:482] T f35a17e1683e41f4b146284f5cf09c58 P eb1fca62dcb34ed9b32188ccb19f0543 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:20.938982 25801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1765495856a47f09ddbb303f76c635f" candidate_uuid: "eb1fca62dcb34ed9b32188ccb19f0543" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "606d49e053a346e486f48cb31832b815"
I20260807 11:11:20.855477 30884 raft_consensus.cc:3037] T 2ff0e05310b145358043bba0a9a8ba11 P 606d49e053a346e486f48cb31832b815 [term 6 FOLLOWER]: Advancing to term 7
I20260807 11:11:20.925777 30270 raft_consensus.cc:3037] T f5e378b8a19447d69c6c9433f071086e P 4fd6381544574ddca559ce63b9243efc [term 8 FOLLOWER]: Advancing to term 9
I20260807 11:11:20.922715 30695 raft_consensus.cc:2781] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 11:11:20.939514 30922 raft_consensus.cc:482] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:11:20.949190 30922 raft_consensus.cc:3037] T 84f78f4141b342d9b7e397096f34af59 P b22d69db45194908ad3d345eba9badd9 [term 7 FOLLOWER]: Advancing to term 8
I20260807 11:11:20.953028 30695 raft_consensus.cc:686] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [term 5 LEADER]: Becoming Leader. State: Replica: eb1fca62dcb34ed9b32188ccb19f0543, State: Running, Role: LEADER
I20260807 11:11:20.954748 30695 consensus_queue.cc:237] T 7be451f12275472a9c2c53877d9d1a0a P eb1fca62dcb34ed9b32188ccb19f0543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "606d49e053a346e486f48cb31832b815" member_type: VOTER last_known_addr { host: "127.23.120.129"